[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 518825699 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524592K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002008] Switched APIC routing to physical x2apic. [ 0.003026] kvm-guest: setup PV IPIs [ 0.006864] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.010098] LSM: Security Framework initializing [ 0.011072] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013091] *** VALIDATE selinux *** [ 0.019097] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023750] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027119] *** VALIDATE tmpfs *** [ 0.029194] *** VALIDATE proc *** [ 0.030000] *** VALIDATE cgroup *** [ 0.030000] *** VALIDATE cgroup2 *** [ 0.030284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033030] Spectre V2 : User space: Vulnerable [ 0.034014] Speculative Store Bypass: Vulnerable [ 0.037389] debug: unmapping init [mem 0xffffffffaae59000-0xffffffffaae60fff] [ 0.039167] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040687] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041023] ... version: 2 [ 0.042013] ... bit width: 48 [ 0.043012] ... generic registers: 4 [ 0.044014] ... value mask: 0000ffffffffffff [ 0.045015] ... max period: 00007fffffffffff [ 0.046016] ... fixed-purpose events: 3 [ 0.047011] ... event mask: 000000070000000f [ 0.049190] rcu: Hierarchical SRCU implementation. [ 0.051551] smp: Bringing up secondary CPUs ... [ 0.052628] x86: Booting SMP configuration: [ 0.053032] .... node #0, CPUs: #1 #2 #3 [ 0.060124] smp: Brought up 1 node, 4 CPUs [ 0.062016] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.204028] node 0 deferred pages initialised in 139ms [ 0.208195] devtmpfs: initialized [ 0.209323] x86/mm: Memory block size: 128MB [ 0.213203] gcov: version magic: 0x41383552 [ 0.217548] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.221165] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.224445] pinctrl core: initialized pinctrl subsystem [ 0.225223] [ 0.225686] ************************************************************* [ 0.228020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.230010] ** ** [ 0.233018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.234000] ** ** [ 0.234000] ** This means that this kernel is built to expose internal ** [ 0.236017] ** IOMMU data structures, which may compromise security on ** [ 0.238011] ** your system. ** [ 0.239009] ** ** [ 0.241014] ** If you see this message and you are not debugging the ** [ 0.242009] ** kernel, report this immediately to your vendor! ** [ 0.244017] ** ** [ 0.247017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249011] ************************************************************* [ 0.251800] NET: Registered protocol family 16 [ 0.253432] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.255037] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.257059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.260135] cpuidle: using governor menu [ 0.261640] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.263566] PCI: Using configuration type 1 for base access [ 0.266149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.276067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.277073] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.280183] cryptd: max_cpu_qlen set to 1000 [ 0.283201] ACPI: Added _OSI(Module Device) [ 0.285021] ACPI: Added _OSI(Processor Device) [ 0.286015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.288020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.293991] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.298570] ACPI: Interpreter enabled [ 0.299000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.299000] ACPI: Using IOAPIC for interrupt routing [ 0.301105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.304354] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.315187] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.317042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.320016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.324083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.329839] acpiphp: Slot [2] registered [ 0.331148] acpiphp: Slot [3] registered [ 0.333119] acpiphp: Slot [4] registered [ 0.334091] acpiphp: Slot [5] registered [ 0.336140] acpiphp: Slot [6] registered [ 0.337142] acpiphp: Slot [7] registered [ 0.339134] acpiphp: Slot [8] registered [ 0.340111] acpiphp: Slot [9] registered [ 0.342162] acpiphp: Slot [10] registered [ 0.344156] acpiphp: Slot [11] registered [ 0.346135] acpiphp: Slot [12] registered [ 0.348176] acpiphp: Slot [13] registered [ 0.350141] acpiphp: Slot [14] registered [ 0.352137] acpiphp: Slot [15] registered [ 0.354142] acpiphp: Slot [16] registered [ 0.355189] acpiphp: Slot [17] registered [ 0.357124] acpiphp: Slot [18] registered [ 0.359120] acpiphp: Slot [19] registered [ 0.360128] acpiphp: Slot [20] registered [ 0.362106] acpiphp: Slot [21] registered [ 0.363092] acpiphp: Slot [22] registered [ 0.365096] acpiphp: Slot [23] registered [ 0.366083] acpiphp: Slot [24] registered [ 0.367089] acpiphp: Slot [25] registered [ 0.368068] acpiphp: Slot [26] registered [ 0.369105] acpiphp: Slot [27] registered [ 0.371137] acpiphp: Slot [28] registered [ 0.372000] acpiphp: Slot [29] registered [ 0.372000] acpiphp: Slot [30] registered [ 0.373131] acpiphp: Slot [31] registered [ 0.374065] PCI host bridge to bus 0000:00 [ 0.375021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.378023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.380023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.382025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.385025] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.387023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.389176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.393588] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.397419] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.406027] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.411965] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.415046] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.417031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.421025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.423537] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.426735] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.429054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.431687] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.435018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.448027] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.452845] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.458000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.465021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.475027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.488029] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.499473] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.506032] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.513019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.530024] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.540310] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.547019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.554025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.569035] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.576785] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.583020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.587022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.601027] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.612827] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.621014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.626015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.646016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.657510] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.667018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.676018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.694019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.703622] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.710434] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.712361] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.714382] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.717251] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.722072] iommu: Default domain type: Passthrough [ 0.723578] SCSI subsystem initialized [ 0.725194] ACPI: bus type USB registered [ 0.727168] usbcore: registered new interface driver usbfs [ 0.728116] usbcore: registered new interface driver hub [ 0.730111] usbcore: registered new device driver usb [ 0.731200] pps_core: LinuxPPS API ver. 1 registered [ 0.733017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.737083] PTP clock support registered [ 0.740053] EDAC MC: Ver: 3.0.0 [ 0.742199] PCI: Using ACPI for IRQ routing [ 0.744952] NetLabel: Initializing [ 0.746014] NetLabel: domain hash size = 128 [ 0.748014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.750077] NetLabel: unlabeled traffic allowed by default [ 0.752110] vgaarb: loaded [ 0.754019] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.755015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.764091] clocksource: Switched to clocksource kvm-clock [ 0.871834] VFS: Disk quotas dquot_6.6.0 [ 0.873459] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.876076] *** VALIDATE ramfs *** [ 0.877401] *** VALIDATE hugetlbfs *** [ 0.878812] pnp: PnP ACPI init [ 0.881373] pnp: PnP ACPI: found 6 devices [ 0.905744] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.909363] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.911429] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.913671] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.916212] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.918530] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.921132] NET: Registered protocol family 2 [ 0.923627] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.928476] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932752] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.938699] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.943233] TCP: Hash tables configured (established 65536 bind 65536) [ 0.946451] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.949758] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.954087] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.957751] NET: Registered protocol family 1 [ 0.961166] RPC: Registered named UNIX socket transport module. [ 0.963707] RPC: Registered udp transport module. [ 0.967504] RPC: Registered tcp transport module. [ 0.969480] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.971899] NET: Registered protocol family 44 [ 0.973409] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.975709] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.981268] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.983606] PCI: CLS 0 bytes, default 64 [ 0.985137] Unpacking initramfs... [ 2.394465] debug: unmapping init [mem 0xffff8b8e7cc54000-0xffff8b8e7ffbffff] [ 2.398677] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.401221] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.404145] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.935382] Initialise system trusted keyrings [ 2.936932] Key type blacklist registered [ 2.939726] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.953577] zbud: loaded [ 2.956468] *** VALIDATE nfs *** [ 2.957520] *** VALIDATE nfs4 *** [ 2.960607] pstore: using deflate compression [ 2.964178] Platform Keyring initialized [ 3.075954] NET: Registered protocol family 38 [ 3.077199] Key type asymmetric registered [ 3.078245] Asymmetric key parser 'x509' registered [ 3.080198] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.082711] io scheduler mq-deadline registered [ 3.083881] io scheduler kyber registered [ 3.084995] io scheduler bfq registered [ 3.087523] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.089527] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.091561] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.095555] ACPI: Power Button [PWRF] [ 3.191349] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.284457] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.474245] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.576325] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.777315] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.808836] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.842681] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.847413] Non-volatile memory driver v1.3 [ 3.848915] Linux agpgart interface v0.103 [ 3.876445] virtio_blk virtio1: [vda] 132416 512-byte logical blocks (67.8 MB/64.7 MiB) [ 3.879270] vda: detected capacity change from 0 to 67796992 [ 3.896416] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.899826] vdb: detected capacity change from 0 to 1073741824 [ 3.922482] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.929504] vdc: detected capacity change from 0 to 2621440000 [ 3.945170] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.948399] vdd: detected capacity change from 0 to 2621440000 [ 3.962982] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.966375] vde: detected capacity change from 0 to 4294967296 [ 3.981469] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.984546] vdf: detected capacity change from 0 to 4294967296 [ 3.991898] libphy: Fixed MDIO Bus: probed [ 4.001514] usbcore: registered new interface driver usbserial_generic [ 4.004252] usbserial: USB Serial support registered for generic [ 4.007269] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.011749] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.013544] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.015831] mousedev: PS/2 mouse device common for all mice [ 4.021025] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.024052] rtc_cmos 00:05: RTC can wake from S4 [ 4.030643] rtc_cmos 00:05: registered as rtc0 [ 4.033206] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.033542] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.033579] intel_pstate: CPU model not supported [ 4.035604] hid: raw HID events driver (C) Jiri Kosina [ 4.042173] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.043744] usbcore: registered new interface driver usbhid [ 4.053929] usbhid: USB HID core driver [ 4.055271] drop_monitor: Initializing network drop monitor service [ 4.057553] Initializing XFRM netlink socket [ 4.059358] NET: Registered protocol family 10 [ 4.062744] Segment Routing with IPv6 [ 4.064048] NET: Registered protocol family 17 [ 4.065866] mpls_gso: MPLS GSO support [ 4.071974] RAS: Correctable Errors collector initialized. [ 4.073695] AVX version of gcm_enc/dec engaged. [ 4.075084] AES CTR mode by8 optimization enabled [ 4.189506] sched_clock: Marking stable (4189489218, 0)->(5137961966, -948472748) [ 4.193375] registered taskstats version 1 [ 4.195519] Loading compiled-in X.509 certificates [ 4.197642] zswap: loaded using pool lzo/zbud [ 4.218027] Key type big_key registered [ 4.234727] Key type encrypted registered [ 4.236255] ima: No TPM chip found, activating TPM-bypass! [ 4.238528] ima: Allocated hash algorithm: sha1 [ 4.239888] ima: No architecture policies found [ 4.241530] evm: Initialising EVM extended attributes: [ 4.243456] evm: security.selinux [ 4.244782] evm: security.ima [ 4.245908] evm: security.capability [ 4.247270] evm: HMAC attrs: 0x1 [ 4.250260] rtc_cmos 00:05: setting system clock to 2025-07-25 10:39:43 UTC (1753439983) [ 4.256486] debug: unmapping init [mem 0xffffffffabe03000-0xffffffffabffffff] [ 4.262341] debug: unmapping init [mem 0xffffffffaab82000-0xffffffffaae58fff] [ 4.272120] Write protecting the kernel read-only data: 28672k [ 4.276277] debug: unmapping init [mem 0xffffffffa9203000-0xffffffffa93fffff] [ 4.280330] debug: unmapping init [mem 0xffffffffa9b14000-0xffffffffa9bfffff] [ 4.315783] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.324898] systemd[1]: Detected virtualization kvm. [ 4.326801] systemd[1]: Detected architecture x86-64. [ 4.328602] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.360682] systemd[1]: No hostname configured. [ 4.363847] systemd[1]: Set hostname to . [ 4.365485] random: systemd: uninitialized urandom read (16 bytes read) [ 4.369895] systemd[1]: Initializing machine ID from random generator. [ 4.429608] random: ln: uninitialized urandom read (6 bytes read) [ 4.522818] random: systemd: uninitialized urandom read (16 bytes read) [ 4.526105] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.530438] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.538758] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.310811] device-mapper: uevent: version 1.0.3 [ 5.314758] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.187374] virtio_net virtio0 ens2: renamed from eth0 [ 6.209877] random: fast init done [ 6.421024] scsi host0: ata_piix [ 6.474191] scsi host1: ata_piix [ 6.475886] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.478519] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.763371] random: crng init done [ 10.767382] random: 7 urandom warning(s) missed due to ratelimiting [ 10.773897] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.395274] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.188627] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.669122] SELinux: Disabled at runtime. [ 14.757670] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.772679] systemd[1]: Detected virtualization kvm. [ 14.774614] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.862949] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.869779] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.880667] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.886751] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.889422] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.902386] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.917743] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. [ 16.248347] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 16.990348] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.882675] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.004395] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.165080] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.218451] EDAC sbridge: Ver: 1.1.2 [ 22.424980] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 22.874951] NFS: Registering the id_resolver key type [ 22.877305] Key type id_resolver registered [ 22.880911] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg349-server login: [ 67.257480] libcfs: loading out-of-tree module taints kernel. [ 67.305820] Key type ._llcrypt registered [ 67.309681] Key type .llcrypt registered [ 67.469171] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 85.988528] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 87.581407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 87.605351] alg: No test for adler32 (adler32-zlib) [ 89.132474] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 89.901290] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 89.907060] LNet: Accept secure, port 988 [ 91.719126] Key type lgssc registered [ 93.142238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.533821] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 110.564421] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.016106] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 126.069857] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.665452] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.904538] hrtimer: interrupt took 4625245 ns [ 149.431904] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 161.652678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.718322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.738754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.912181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 162.950320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.011037] Lustre: lustre-MDT0000: new disk, initializing [ 163.085463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.114603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 166.617966] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.893607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 178.009727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 178.063536] Lustre: 6479:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.086379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.090223] Lustre: Skipped 1 previous similar message [ 178.150616] Lustre: lustre-MDT0001: new disk, initializing [ 178.241638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.272594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.290040] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.088795] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.822053] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 192.880180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.974399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 193.208322] Lustre: lustre-OST0000: new disk, initializing [ 193.211262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.287737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.889783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.903769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.954221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.946547] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.628390] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 210.703979] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 210.778740] Lustre: lustre-OST0001: new disk, initializing [ 210.783623] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 210.866831] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.761402] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.142298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.150233] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.240182] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 226.904289] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.701274] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 242.289548] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.382539] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing yml_node [ 251.371463] Lustre: DEBUG MARKER: Client: 2.16.56.57 [ 254.090966] Lustre: DEBUG MARKER: MDS: 2.16.56.57 [ 256.103279] Lustre: DEBUG MARKER: OSS: 2.16.56.57 [ 257.479748] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 25 06:43:55 EDT 2025 [ 272.649623] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 274.291958] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.783786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.788394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.801038] Lustre: Skipped 2 previous similar messages [ 290.798994] Lustre: Skipped 3 previous similar messages [ 294.812209] LustreError: 12147:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 294.886569] Lustre: server umount lustre-MDT0000 complete [ 297.642462] LustreError: 10278:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440276 with bad export cookie 17744880193624368767 [ 297.642821] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 297.652465] LustreError: 10278:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 301.027201] LustreError: 6485:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.030535] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.030971] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.030977] Lustre: Skipped 1 previous similar message [ 301.043644] LustreError: 6485:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 301.057092] Lustre: Skipped 1 previous similar message [ 302.049648] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 302.049734] LustreError: 10395:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.053266] Lustre: Skipped 2 previous similar messages [ 302.064482] LustreError: 10395:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 303.829342] LustreError: 12349:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 303.831805] LustreError: 12349:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 303.983823] Lustre: server umount lustre-MDT0001 complete [ 306.751667] LustreError: 12550:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 306.757783] LustreError: 12550:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 306.806722] Lustre: server umount lustre-OST0000 complete [ 309.452123] LustreError: 12750:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 309.455344] LustreError: 12750:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 309.623849] Lustre: server umount lustre-OST0001 complete [ 313.700626] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 319.474582] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 325.100810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 329.054606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 333.067483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 337.075387] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.768920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.804075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.945896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 340.967973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 341.013804] Lustre: lustre-MDT0000: new disk, initializing [ 341.081431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 341.094211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 343.308210] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 349.195499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 349.234144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 349.289860] Lustre: 14918:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 349.338465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 349.341195] Lustre: Skipped 1 previous similar message [ 349.396766] Lustre: lustre-MDT0001: new disk, initializing [ 349.471590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 349.501385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 349.506992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 351.610992] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 355.825925] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 357.627122] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 361.328414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 361.387211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 363.624924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 363.706939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 365.078439] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 370.122688] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 372.324527] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 372.450890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 374.512408] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 374.632973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 379.363714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 379.363940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 379.367173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 379.373243] Lustre: Skipped 1 previous similar message [ 381.752817] LustreError: 17825:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 381.765317] LustreError: 17825:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 381.845476] Lustre: server umount lustre-OST0000 complete [ 384.482144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.484153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 384.488203] Lustre: Skipped 1 previous similar message [ 384.490210] Lustre: Skipped 1 previous similar message [ 394.720135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 394.725302] Lustre: Skipped 4 previous similar messages [ 398.304509] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 398.352932] LustreError: 18027:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 398.354973] LustreError: 18027:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 398.390271] Lustre: server umount lustre-MDT0000 complete [ 398.816443] LustreError: 15848:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.823775] LustreError: 15848:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 400.160366] LustreError: 14910:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440379 with bad export cookie 17744880193624370300 [ 400.164214] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 400.166807] LustreError: 14910:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 400.361577] Lustre: server umount lustre-MDT0001 complete [ 402.037797] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 06:46:20 (1753440380) [ 405.850579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.112550] LustreError: 18753:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.169239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.171821] Lustre: Skipped 1 previous similar message [ 408.341327] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 411.616725] LustreError: 18753:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.759326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 414.786845] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 417.191190] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 418.892459] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 421.995849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 424.352214] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 426.417225] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 436.708189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 436.709784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.711757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 436.720966] Lustre: Skipped 1 previous similar message [ 437.039126] LustreError: 20977:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 437.042255] LustreError: 20977:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 437.122509] Lustre: server umount lustre-OST0000 complete [ 441.053181] Lustre: server umount lustre-MDT0000 complete [ 441.827469] LustreError: 19458:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.833747] LustreError: 19458:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 442.767415] LustreError: 18734:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440422 with bad export cookie 17744880193624371182 [ 442.771440] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 442.773219] LustreError: 18734:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 442.943487] Lustre: server umount lustre-MDT0001 complete [ 448.367302] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 449.720896] Key type lgssc unregistered [ 449.916803] LNet: 21906:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.981591] LNet: Removed LNI 192.168.203.149@tcp [ 451.415487] Key type .llcrypt unregistered [ 451.417160] Key type ._llcrypt unregistered [ 461.459119] Key type ._llcrypt registered [ 461.461519] Key type .llcrypt registered [ 461.506582] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 461.867995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 461.894359] alg: No test for adler32 (adler32-zlib) [ 462.761344] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 462.855679] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 462.857910] LNet: Accept secure, port 988 [ 464.447175] Key type lgssc registered [ 464.902305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 468.543261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 468.553867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 469.788479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.774849] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 476.888399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 477.112480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 479.238290] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 482.010080] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 483.908896] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 487.627434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 487.866244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.186759] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 494.212174] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 495.027813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 510.433483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.437162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.443793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.483141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 512.483979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.487025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.603039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 517.607067] Lustre: Skipped 1 previous similar message [ 522.207160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 522.228037] LustreError: 25934:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 522.298626] Lustre: server umount lustre-OST0000 complete [ 527.841107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.841764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 527.845822] Lustre: Skipped 1 previous similar message [ 527.848385] Lustre: Skipped 1 previous similar message [ 532.960076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 532.962812] Lustre: Skipped 2 previous similar messages [ 538.079699] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 538.118237] LustreError: 26136:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 538.121117] LustreError: 26136:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 538.151884] Lustre: server umount lustre-MDT0000 complete [ 539.645108] LustreError: 23244:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440518 with bad export cookie 1834018520845064774 [ 539.646473] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.650226] LustreError: 23244:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 539.722711] LustreError: 26338:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 539.725710] LustreError: 26338:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 539.819549] Lustre: server umount lustre-MDT0001 complete [ 544.870289] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 545.988878] Key type lgssc unregistered [ 546.130580] LNet: 26864:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.173589] LNet: Removed LNI 192.168.203.149@tcp [ 547.518112] Key type .llcrypt unregistered [ 547.519270] Key type ._llcrypt unregistered [ 557.530227] Key type ._llcrypt registered [ 557.531790] Key type .llcrypt registered [ 557.577296] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 558.015273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 558.028726] alg: No test for adler32 (adler32-zlib) [ 558.893334] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 558.986297] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 558.988591] LNet: Accept secure, port 988 [ 560.575177] Key type lgssc registered [ 561.008429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 564.177796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 564.188959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 565.376593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.967683] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 570.674419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 570.858474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 572.541466] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 574.832729] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 576.359299] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 581.404504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 581.548312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.661119] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 585.888275] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 590.762214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 601.061146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.062099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 601.062731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 601.066705] Lustre: Skipped 1 previous similar message [ 601.075530] LustreError: Skipped 1 previous similar message [ 605.232116] LustreError: 30817:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 605.302451] Lustre: server umount lustre-OST0000 complete [ 611.296668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.297301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 611.301023] Lustre: Skipped 1 previous similar message [ 611.303878] Lustre: Skipped 1 previous similar message [ 612.989315] LustreError: 31018:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 612.992205] LustreError: 31018:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 613.006290] LustreError: 31018:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000adbe6188) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 615.073106] LustreError: 31018:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 615.076275] LustreError: 31018:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 615.090686] Lustre: server umount lustre-MDT0000 complete [ 616.368726] LustreError: 28205:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440595 with bad export cookie 12576048445152601145 [ 616.372076] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.374338] LustreError: 28205:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 616.528526] Lustre: server umount lustre-MDT0001 complete [ 621.432182] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 622.577332] Key type lgssc unregistered [ 622.719667] LNet: 31747:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.782405] LNet: Removed LNI 192.168.203.149@tcp [ 624.152261] Key type .llcrypt unregistered [ 624.154279] Key type ._llcrypt unregistered [ 633.324520] Key type ._llcrypt registered [ 633.326082] Key type .llcrypt registered [ 633.368406] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 633.678411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.685616] alg: No test for adler32 (adler32-zlib) [ 634.534779] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 634.616894] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 634.618500] LNet: Accept secure, port 988 [ 636.199169] Key type lgssc registered [ 636.590470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 639.348298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 639.355749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.532928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 641.923489] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 644.980526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 645.124201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 646.472820] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 648.429812] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 649.823342] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 652.371790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 652.508826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 654.611152] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 656.539615] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 665.831711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 671.202203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 671.202717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 671.204285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 671.205682] LustreError: Skipped 1 previous similar message [ 671.211275] Lustre: Skipped 1 previous similar message [ 672.171143] LustreError: 35358:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 672.237649] Lustre: server umount lustre-OST0000 complete [ 675.810094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 675.813271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.827095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 675.829593] Lustre: Skipped 1 previous similar message [ 675.860475] LustreError: 35559:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 675.863587] LustreError: 35559:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 675.890437] Lustre: server umount lustre-MDT0000 complete [ 676.322851] LustreError: 33107:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.330556] LustreError: 33107:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 677.301845] LustreError: 33087:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440656 with bad export cookie 12929895456568525254 [ 677.304834] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.306857] LustreError: 33087:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 677.377452] LustreError: 35762:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 677.380216] LustreError: 35762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 677.467760] Lustre: server umount lustre-MDT0001 complete [ 682.369293] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 683.523176] Key type lgssc unregistered [ 683.669489] LNet: 36288:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.709342] LNet: Removed LNI 192.168.203.149@tcp [ 685.023963] Key type .llcrypt unregistered [ 685.024977] Key type ._llcrypt unregistered [ 693.803412] Key type ._llcrypt registered [ 693.805111] Key type .llcrypt registered [ 693.850052] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 694.277713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 694.288339] alg: No test for adler32 (adler32-zlib) [ 695.146646] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 695.224420] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 695.226805] LNet: Accept secure, port 988 [ 696.807217] Key type lgssc registered [ 697.164345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 699.720740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 699.726951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.869438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.133442] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 704.900250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 705.017380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 706.270297] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 707.970968] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 709.120765] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 711.554217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 711.668148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 713.617179] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 715.453420] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 725.990548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 731.104913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 731.105347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.108476] LustreError: Skipped 1 previous similar message [ 731.115236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 731.118873] Lustre: Skipped 1 previous similar message [ 734.446161] LustreError: 39894:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 734.511257] Lustre: server umount lustre-OST0000 complete [ 750.817798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.818664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 750.822598] Lustre: Skipped 2 previous similar messages [ 750.825429] Lustre: Skipped 1 previous similar message [ 756.192050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.194169] Lustre: Skipped 2 previous similar messages [ 761.312389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 761.314650] Lustre: Skipped 1 previous similar message [ 762.335131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 762.370582] LustreError: 40269:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 762.373262] LustreError: 40269:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 762.408525] Lustre: server umount lustre-MDT0000 complete [ 763.977316] LustreError: 40018:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440743 with bad export cookie 15697867496778719003 [ 763.980288] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 763.981609] LustreError: 40018:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 770.121251] LustreError: 40473:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 770.124905] LustreError: 40473:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 770.218955] Lustre: server umount lustre-MDT0001 complete [ 775.637455] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 776.954389] Key type lgssc unregistered [ 777.121884] LNet: 40999:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.149502] LNet: Removed LNI 192.168.203.149@tcp [ 778.507463] Key type .llcrypt unregistered [ 778.509260] Key type ._llcrypt unregistered [ 788.273428] Key type ._llcrypt registered [ 788.274514] Key type .llcrypt registered [ 788.315312] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 788.644113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 788.659089] alg: No test for adler32 (adler32-zlib) [ 789.551320] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 789.664200] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 789.668221] LNet: Accept secure, port 988 [ 791.271172] Key type lgssc registered [ 791.660448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 794.666400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 794.675282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.875621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.523349] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 800.996584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 801.149689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 802.714414] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 804.858774] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 806.177454] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 808.569022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 808.709184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 810.849730] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 812.937318] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 821.035131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 826.337520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.338678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 826.342680] Lustre: Skipped 2 previous similar messages [ 826.347210] Lustre: Skipped 2 previous similar messages [ 828.552153] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.49@tcp (stopping) [ 831.456134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 831.458385] Lustre: Skipped 2 previous similar messages [ 832.124215] LustreError: 44607:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 832.152933] Lustre: server umount lustre-MDT0000 complete [ 833.588530] LustreError: 42340:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753440812 with bad export cookie 828614164794671597 [ 833.592925] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 833.593442] LustreError: 42340:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 833.661326] LustreError: 43605:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.667813] LustreError: 43605:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 833.670219] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.49@tcp (stopping) [ 836.576095] LustreError: 43605:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 836.576359] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.624069] LustreError: 43605:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 838.624298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 838.631529] Lustre: Skipped 1 previous similar message [ 839.743595] LustreError: 44808:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 839.746026] LustreError: 44808:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 839.827926] Lustre: server umount lustre-MDT0001 complete [ 866.271127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 866.286785] LustreError: 45032:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 866.289677] LustreError: 45032:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 866.367577] Lustre: server umount lustre-OST0000 complete [ 873.598745] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 874.722738] Key type lgssc unregistered [ 874.863722] LNet: 45865:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.878358] LNet: Removed LNI 192.168.203.149@tcp [ 876.212864] Key type .llcrypt unregistered [ 876.214147] Key type ._llcrypt unregistered [ 886.452100] Key type ._llcrypt registered [ 886.453563] Key type .llcrypt registered [ 886.492044] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 886.823439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.871894] alg: No test for adler32 (adler32-zlib) [ 887.748597] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 887.847924] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 887.850585] LNet: Accept secure, port 988 [ 889.439191] Key type lgssc registered [ 889.850886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 892.729508] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 892.736032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 909.215365] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 915.359195] LustreError: 47198:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 945.055220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 947.160838] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 949.176699] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1117.209417] LustreError: 47903:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1117.376201] Lustre: server umount lustre-OST0000 complete [ 1130.088124] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1132.361134] Key type lgssc unregistered [ 1132.582909] LNet: 48734:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.608208] LNet: Removed LNI 192.168.203.149@tcp [ 1134.295986] Key type .llcrypt unregistered [ 1134.298447] Key type ._llcrypt unregistered [ 1151.051978] Key type ._llcrypt registered [ 1151.053735] Key type .llcrypt registered [ 1151.135966] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1151.689758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1151.711664] alg: No test for adler32 (adler32-zlib) [ 1152.693884] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1152.829514] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1152.832793] LNet: Accept secure, port 988 [ 1154.463469] Key type lgssc registered [ 1155.242032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1160.834390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1160.860620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.238642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.782837] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1171.128139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1171.494719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1174.422872] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1178.390823] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1181.247250] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1186.838528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1187.201279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1192.210825] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1197.212511] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1201.638381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.658406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1205.301195] LustreError: 52320:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1205.516078] Lustre: server umount lustre-OST0000 complete [ 1211.808292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.826788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1211.831468] Lustre: Skipped 1 previous similar message [ 1214.552904] LustreError: 52522:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1214.562420] LustreError: 52522:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1214.647878] Lustre: server umount lustre-MDT0000 complete [ 1216.992674] LustreError: 50095:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.020887] LustreError: 50095:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1217.502564] LustreError: 50077:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441196 with bad export cookie 15736989567898925432 [ 1217.503240] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1217.512597] LustreError: 50077:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1217.663269] LustreError: 52723:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1217.675665] LustreError: 52723:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1217.889546] Lustre: server umount lustre-MDT0001 complete [ 1226.138914] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1227.930813] Key type lgssc unregistered [ 1228.144311] LNet: 53250:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.163803] LNet: Removed LNI 192.168.203.149@tcp [ 1229.701327] Key type .llcrypt unregistered [ 1229.703754] Key type ._llcrypt unregistered [ 1243.811782] Key type ._llcrypt registered [ 1243.813346] Key type .llcrypt registered [ 1243.873767] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1244.328464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1244.356192] alg: No test for adler32 (adler32-zlib) [ 1245.410676] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1245.572747] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1245.575566] LNet: Accept secure, port 988 [ 1247.239166] Key type lgssc registered [ 1248.165329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1255.444090] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1255.463739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1272.096651] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1278.240815] LustreError: 54583:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 1309.023185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1313.056517] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1317.172959] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1322.583305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.053263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1323.099400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1325.853739] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1328.105126] LustreError: 54609:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.113449] LustreError: 54609:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1332.566281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1332.984160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1336.344091] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1341.439922] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1344.455702] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1347.564000] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.566978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1347.578801] Lustre: Skipped 1 previous similar message [ 1352.680903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1352.683679] Lustre: Skipped 2 previous similar messages [ 1357.795396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1357.798477] Lustre: Skipped 1 previous similar message [ 1360.351175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1360.393149] LustreError: 56861:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1360.511421] Lustre: server umount lustre-OST0000 complete [ 1383.392971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.396442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1383.414990] Lustre: Skipped 1 previous similar message [ 1386.267377] LustreError: 57244:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1386.270345] LustreError: 57244:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1386.341953] Lustre: server umount lustre-MDT0000 complete [ 1388.517491] LustreError: 55525:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.526934] LustreError: 55525:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1389.383768] LustreError: 56863:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441368 with bad export cookie 7821854333513470488 [ 1389.390115] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1389.398940] LustreError: 56863:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1395.617367] LustreError: 57445:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1395.630932] LustreError: 57445:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1395.868568] Lustre: server umount lustre-MDT0001 complete [ 1405.191207] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1407.851935] Key type lgssc unregistered [ 1408.121538] LNet: 57972:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.193498] LNet: Removed LNI 192.168.203.149@tcp [ 1409.795619] Key type .llcrypt unregistered [ 1409.798467] Key type ._llcrypt unregistered [ 1426.605623] Key type ._llcrypt registered [ 1426.607241] Key type .llcrypt registered [ 1426.698372] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1427.420350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1427.491114] alg: No test for adler32 (adler32-zlib) [ 1428.512055] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1428.671172] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1428.674731] LNet: Accept secure, port 988 [ 1430.327158] Key type lgssc registered [ 1431.080418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1436.743703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1436.767809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.076750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.796112] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1446.977822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1447.265169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1450.097697] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1454.120512] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1456.969536] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1461.899375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1462.108964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.151804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1466.467655] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1470.882846] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1498.086929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.087754] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1498.104890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.183041] LustreError: 61581:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1501.366590] Lustre: server umount lustre-OST0000 complete [ 1508.323089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.324137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.346357] Lustre: Skipped 1 previous similar message [ 1508.348466] Lustre: Skipped 2 previous similar messages [ 1510.687753] LustreError: 61783:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1510.694090] LustreError: 61783:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1510.772801] Lustre: server umount lustre-MDT0000 complete [ 1513.278680] LustreError: 59314:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441492 with bad export cookie 10262599966154400221 [ 1513.279980] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1513.293470] LustreError: 59314:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1513.440249] LustreError: 61986:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1513.444267] LustreError: 61986:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1513.634421] Lustre: server umount lustre-MDT0001 complete [ 1522.371495] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1524.888566] Key type lgssc unregistered [ 1525.126292] LNet: 62512:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.182909] LNet: Removed LNI 192.168.203.149@tcp [ 1526.799618] Key type .llcrypt unregistered [ 1526.801599] Key type ._llcrypt unregistered [ 1551.404359] Key type ._llcrypt registered [ 1551.406328] Key type .llcrypt registered [ 1551.504140] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1552.343795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1552.376475] alg: No test for adler32 (adler32-zlib) [ 1553.367277] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1553.607930] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1553.610966] LNet: Accept secure, port 988 [ 1555.255166] Key type lgssc registered [ 1556.218270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1562.706103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1562.720111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.047300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.905282] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1573.497687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1573.843197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1577.282946] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1581.973913] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1584.698659] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1590.196400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1590.488833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.111432] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1597.912742] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1609.961763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1618.098577] Lustre: Failing over lustre-MDT0000 [ 1618.265301] LustreError: 66411:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1619.938467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1619.944609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.956715] LustreError: 64165:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1619.964732] LustreError: 64165:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1620.325943] LustreError: 66411:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 1620.336027] LustreError: 66411:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1620.361726] Lustre: server umount lustre-MDT0000 complete [ 1620.452849] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.472706] LustreError: 64161:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.488891] LustreError: 64161:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1625.219278] LustreError: 64161:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1627.007358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.097938] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1627.209765] Lustre: *** cfs_fail_loc=135, val=0*** [ 1627.315452] BUG: spinlock bad magic on CPU#2, llog_process_th/66921 [ 1627.319160] lock: 0xffff8b8ec84d70c0, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 1627.326458] CPU: 2 PID: 66921 Comm: llog_process_th Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1627.342091] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 1627.345781] Call Trace: [ 1627.348978] ? dump_stack+0xbb/0x10e [ 1627.351504] ? spin_dump.cold.3+0x32/0x65 [ 1627.354121] ? do_raw_spin_lock+0xce/0x190 [ 1627.355633] ? _raw_spin_lock+0x21/0x30 [ 1627.356805] ? mdd_changelog_fini+0x45/0x3a0 [mdd] [ 1627.358376] ? mdd_process_config+0x368/0x930 [mdd] [ 1627.361592] ? mdt_stack_fini+0x5de/0xe50 [mdt] [ 1627.366551] ? mdt_init0+0x6c0/0x18f0 [mdt] [ 1627.370954] ? _raw_spin_unlock+0x12/0x30 [ 1627.373321] ? mdt_device_alloc+0x118/0x170 [mdt] [ 1627.375301] ? obd_setup+0x2e6/0x700 [obdclass] [ 1627.378045] ? class_setup+0x6f5/0x9f0 [obdclass] [ 1627.380900] ? class_process_config+0x1658/0x2b60 [obdclass] [ 1627.384024] ? class_config_llog_handler+0x8ac/0x1af0 [obdclass] [ 1627.390349] ? __kmalloc+0x1b4/0x4a0 [ 1627.392273] ? class_config_llog_handler+0xc51/0x1af0 [obdclass] [ 1627.394841] ? llog_process_thread+0x13be/0x2220 [obdclass] [ 1627.398048] ? llog_process+0x40/0x40 [obdclass] [ 1627.402739] ? llog_process_thread_daemonize+0xa2/0xe0 [obdclass] [ 1627.405049] ? kthread+0x1d1/0x200 [ 1627.410737] ? set_kthread_struct+0x70/0x70 [ 1627.412710] ? ret_from_fork+0x1f/0x30 [ 1627.418282] LustreError: 66921:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000 failed (-2) [ 1627.423506] LustreError: 66921:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -2 [ 1627.430664] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1627.430664] [ 1627.443618] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1627.455912] LustreError: 66909:0:(tgt_mount.c:1527:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1627.464618] LustreError: 66909:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -2 [ 1627.472633] LustreError: 66909:0:(obd_config.c:832:class_cleanup()) Device 5 not setup [ 1627.486388] Lustre: server umount lustre-MDT0000 complete [ 1627.492795] LustreError: 66909:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1630.343904] LustreError: 66208:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.356272] LustreError: 66208:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1632.263154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.369761] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1632.555206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.609250] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1635.385472] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1635.453401] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1637.865487] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1637.878599] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1637.902327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1642.978448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.983944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1642.989533] Lustre: Skipped 2 previous similar messages [ 1646.646162] LustreError: 67724:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1646.650425] LustreError: 67724:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1646.733609] Lustre: server umount lustre-OST0000 complete [ 1653.217268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1653.220698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1653.228571] Lustre: Skipped 1 previous similar message [ 1653.233485] Lustre: Skipped 1 previous similar message [ 1655.063138] LustreError: 67925:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1655.066630] LustreError: 67925:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1655.159530] Lustre: server umount lustre-MDT0000 complete [ 1657.568862] LustreError: 66412:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441636 with bad export cookie 8782175642015573204 [ 1657.569561] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1657.575151] LustreError: 66412:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1657.868626] Lustre: server umount lustre-MDT0001 complete [ 1665.170181] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1666.702751] Key type lgssc unregistered [ 1666.875934] LNet: 68655:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.941830] LNet: Removed LNI 192.168.203.149@tcp [ 1668.337494] Key type .llcrypt unregistered [ 1668.339242] Key type ._llcrypt unregistered [ 1681.982716] Key type ._llcrypt registered [ 1681.984330] Key type .llcrypt registered [ 1682.071770] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1682.584251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1682.627381] alg: No test for adler32 (adler32-zlib) [ 1683.607480] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1683.782288] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1683.786125] LNet: Accept secure, port 988 [ 1685.415315] Key type lgssc registered [ 1685.917711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1690.276357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1690.287798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.518103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.448181] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1698.048818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1698.300954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1700.307867] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1703.284852] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1705.113747] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1708.858804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1709.066068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1712.292816] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1715.378514] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1716.266160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1728.351158] Lustre: Failing over lustre-MDT0000 [ 1728.474109] LustreError: 72259:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1728.484301] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.488293] LustreError: 71365:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1728.489833] Lustre: Skipped 2 previous similar messages [ 1728.496611] LustreError: 71365:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1728.521541] Lustre: server umount lustre-MDT0000 complete [ 1732.245820] LustreError: 71365:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.537744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.601760] LustreError: 70013:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.609581] LustreError: 70013:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1733.631839] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1733.731216] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1733.801851] LustreError: 72768:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000 failed (-9) [ 1733.805464] LustreError: 72768:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -9 [ 1733.812022] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1733.812022] [ 1733.819801] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1733.829865] LustreError: 72756:0:(tgt_mount.c:1527:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1733.834297] LustreError: 72756:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -9 [ 1733.841593] LustreError: 72756:0:(obd_config.c:832:class_cleanup()) Device 5 not setup [ 1733.856662] Lustre: server umount lustre-MDT0000 complete [ 1733.858889] LustreError: 72756:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1737.347580] LustreError: 70008:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.906320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.976829] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1738.139137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.190157] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1740.414734] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1742.464508] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1743.334117] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1743.353450] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1743.384315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1748.451306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.451409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1748.459211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1748.463266] Lustre: Skipped 1 previous similar message [ 1751.415250] LustreError: 73574:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1751.419419] LustreError: 73574:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1751.513066] Lustre: server umount lustre-OST0000 complete [ 1753.571218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.572479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1753.574203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1753.583864] Lustre: Skipped 2 previous similar messages [ 1758.689792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.694208] Lustre: Skipped 2 previous similar messages [ 1759.574691] LustreError: 73775:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1759.577923] LustreError: 73775:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1759.639041] Lustre: server umount lustre-MDT0000 complete [ 1762.006069] LustreError: 69993:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441741 with bad export cookie 14434354217520413854 [ 1762.007664] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1762.011801] LustreError: 69993:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1762.116226] LustreError: 73977:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1762.118373] LustreError: 73977:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1762.294819] Lustre: server umount lustre-MDT0001 complete [ 1769.616876] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1771.599667] Key type lgssc unregistered [ 1771.831752] LNet: 74504:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.902493] LNet: Removed LNI 192.168.203.149@tcp [ 1773.552778] Key type .llcrypt unregistered [ 1773.554682] Key type ._llcrypt unregistered [ 1780.602719] Key type ._llcrypt registered [ 1780.604504] Key type .llcrypt registered [ 1780.682615] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1780.873623] Key type .llcrypt unregistered [ 1780.875177] Key type ._llcrypt unregistered [ 1795.887705] Key type ._llcrypt registered [ 1795.893280] Key type .llcrypt registered [ 1795.981613] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1796.530892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1796.561743] alg: No test for adler32 (adler32-zlib) [ 1797.503953] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1797.637436] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1797.641784] LNet: Accept secure, port 988 [ 1799.264429] Key type lgssc registered [ 1799.893044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1804.442869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1804.466721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.761657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.224337] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1812.907738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1813.102754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1815.086695] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1817.804894] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1819.820469] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1823.209912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1823.269459] LustreError: 77914:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1823.277284] LustreError: 77914:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1823.417262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.267971] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1829.200887] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1829.867772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 1834.976702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1834.977558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.979352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1834.989316] Lustre: Skipped 1 previous similar message [ 1840.100260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.104612] Lustre: Skipped 2 previous similar messages [ 1841.004892] LustreError: 78554:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1841.101438] Lustre: server umount lustre-OST0000 complete [ 1843.682361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1843.686440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1843.693144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1845.218326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.040207] LustreError: 78755:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1849.042921] LustreError: 78755:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1849.098766] Lustre: server umount lustre-MDT0000 complete [ 1850.336148] LustreError: 77033:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1850.343836] LustreError: 77033:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1850.951418] LustreError: 76307:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441830 with bad export cookie 7367768723718289469 [ 1850.953604] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1850.958990] LustreError: 76307:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1851.045726] LustreError: 78958:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1851.048745] LustreError: 78958:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1851.163323] Lustre: server umount lustre-MDT0001 complete [ 1857.039090] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1858.487898] Key type lgssc unregistered [ 1858.702678] LNet: 79484:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.750288] LNet: Removed LNI 192.168.203.149@tcp [ 1860.269562] Key type .llcrypt unregistered [ 1860.271630] Key type ._llcrypt unregistered [ 1873.713461] Key type ._llcrypt registered [ 1873.715034] Key type .llcrypt registered [ 1873.796778] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1874.325927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1874.390698] alg: No test for adler32 (adler32-zlib) [ 1875.337613] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1875.482285] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1875.487478] LNet: Accept secure, port 988 [ 1877.103188] Key type lgssc registered [ 1877.672038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1881.670579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1881.692325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1882.959046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.645922] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1889.308326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1889.542460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1891.465555] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1894.213304] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1896.106558] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1899.969396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1900.223139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.972780] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1905.543622] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1908.394553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 1918.944233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1918.947583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.953862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.968211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1919.972156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.979049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.988859] Lustre: Skipped 1 previous similar message [ 1920.571693] LustreError: 83096:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1920.702643] Lustre: server umount lustre-OST0000 complete [ 1925.090646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.094700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.122050] Lustre: Skipped 1 previous similar message [ 1930.208369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.211244] Lustre: Skipped 2 previous similar messages [ 1935.328397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.331630] Lustre: Skipped 1 previous similar message [ 1937.375191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1937.436465] LustreError: 83297:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1937.441042] LustreError: 83297:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1937.481570] Lustre: server umount lustre-MDT0000 complete [ 1939.677219] LustreError: 83300:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753441918 with bad export cookie 14765397148942872321 [ 1939.680954] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1939.684766] LustreError: 83300:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1939.773918] LustreError: 83503:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1939.777440] LustreError: 83503:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1939.896172] Lustre: server umount lustre-MDT0001 complete [ 1945.983700] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 1947.634537] Key type lgssc unregistered [ 1947.827772] LNet: 84029:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.906031] LNet: Removed LNI 192.168.203.149@tcp [ 1949.407840] Key type .llcrypt unregistered [ 1949.410027] Key type ._llcrypt unregistered [ 1962.615557] Key type ._llcrypt registered [ 1962.616970] Key type .llcrypt registered [ 1962.686876] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 1963.296862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1963.324993] alg: No test for adler32 (adler32-zlib) [ 1964.314919] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 1964.482602] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 1964.485286] LNet: Accept secure, port 988 [ 1966.119813] Key type lgssc registered [ 1966.710657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1971.166873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1971.190938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.544552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.769610] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1979.968176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1980.180357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1982.392743] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1985.427567] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1987.154161] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1990.526651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1990.694437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1993.567047] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 1996.448324] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1996.841391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2012.131696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2012.135146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.142598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2015.199957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2015.205939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2015.207938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.456119] LustreError: 87641:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2017.553923] Lustre: server umount lustre-OST0000 complete [ 2020.321264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.324182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.327595] Lustre: Skipped 1 previous similar message [ 2020.333658] Lustre: Skipped 2 previous similar messages [ 2025.441359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.444757] Lustre: Skipped 1 previous similar message [ 2025.641162] LustreError: 87843:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2025.647829] LustreError: 87843:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2025.706464] Lustre: server umount lustre-MDT0000 complete [ 2027.947487] LustreError: 85370:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442007 with bad export cookie 4713373469657392965 [ 2027.948569] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2027.953079] LustreError: 85370:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2028.039541] LustreError: 88044:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2028.049070] LustreError: 88044:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2028.178538] Lustre: server umount lustre-MDT0001 complete [ 2034.327081] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2035.812822] Key type lgssc unregistered [ 2035.982443] LNet: 88572:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.029855] LNet: Removed LNI 192.168.203.149@tcp [ 2037.488991] Key type .llcrypt unregistered [ 2037.494064] Key type ._llcrypt unregistered [ 2050.391225] Key type ._llcrypt registered [ 2050.392932] Key type .llcrypt registered [ 2050.453926] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2050.997768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2051.029446] alg: No test for adler32 (adler32-zlib) [ 2051.979336] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2052.108848] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2052.111873] LNet: Accept secure, port 988 [ 2053.735178] Key type lgssc registered [ 2054.334572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2058.608341] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2058.620899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2075.103320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2081.247227] LustreError: 89904:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 2111.967297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.244693] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2118.347497] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2121.277460] LustreError: 90754:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2121.356817] Lustre: server umount lustre-OST0000 complete [ 2125.644538] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 07:15:03 (1753442103) [ 2129.205570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.444725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.640949] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2134.496517] LustreError: 91425:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.505993] LustreError: 91425:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2136.367435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2136.578242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2138.894929] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2142.087919] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2143.987142] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2147.531346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2147.722909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.424969] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2153.365446] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2153.902708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2169.313209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2169.319628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.330765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.360668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2171.362762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2171.365058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.340550] LustreError: 93795:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2174.349053] LustreError: 93795:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2174.487111] Lustre: server umount lustre-OST0000 complete [ 2177.504317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2177.508095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.514190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2177.516687] Lustre: Skipped 1 previous similar message [ 2181.600465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.601348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2181.610168] Lustre: Skipped 1 previous similar message [ 2182.666525] LustreError: 93996:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2182.670484] LustreError: 93996:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2182.713816] Lustre: server umount lustre-MDT0000 complete [ 2184.517558] LustreError: 91405:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442163 with bad export cookie 3801000006178999052 [ 2184.520552] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2184.522767] LustreError: 91405:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2184.698683] Lustre: server umount lustre-MDT0001 complete [ 2190.146848] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2191.384446] Key type lgssc unregistered [ 2191.544496] LNet: 94725:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2192.615347] LNet: Removed LNI 192.168.203.149@tcp [ 2193.022523] Key type .llcrypt unregistered [ 2193.024424] Key type ._llcrypt unregistered [ 2203.769472] Key type ._llcrypt registered [ 2203.771096] Key type .llcrypt registered [ 2203.817453] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2204.184187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2204.241954] alg: No test for adler32 (adler32-zlib) [ 2205.125320] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2205.232946] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2205.235848] LNet: Accept secure, port 988 [ 2206.831121] Key type lgssc registered [ 2207.316426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2211.076837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2211.091783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.322365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.079869] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2218.103515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2218.322271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2220.221289] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2223.085614] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2224.769776] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2227.903528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2228.080043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2230.701891] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2233.063160] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2237.290585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2242.907469] LustreError: 98309:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2242.987437] Lustre: server umount lustre-OST0000 complete [ 2247.648927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.653904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.657062] Lustre: Skipped 1 previous similar message [ 2252.768917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2252.771732] Lustre: Skipped 2 previous similar messages [ 2254.822696] LustreError: 98509:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2254.826848] LustreError: 98509:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2254.873192] Lustre: server umount lustre-MDT0000 complete [ 2257.012967] LustreError: 96066:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442236 with bad export cookie 14784931028950103986 [ 2257.014981] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2257.018052] LustreError: 96066:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2267.621231] LustreError: 98714:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2267.624588] LustreError: 98714:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2267.734903] Lustre: server umount lustre-MDT0001 complete [ 2274.378060] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2276.151689] Key type lgssc unregistered [ 2276.390613] LNet: 99242:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2277.414485] LNet: Removed LNI 192.168.203.149@tcp [ 2277.928185] Key type .llcrypt unregistered [ 2277.929577] Key type ._llcrypt unregistered [ 2287.983635] Key type ._llcrypt registered [ 2287.985681] Key type .llcrypt registered [ 2288.045486] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2288.521784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2288.540383] alg: No test for adler32 (adler32-zlib) [ 2289.478132] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2289.595689] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2289.598803] LNet: Accept secure, port 988 [ 2291.199187] Key type lgssc registered [ 2291.734980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2295.594153] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2295.609032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2312.095342] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2318.239501] LustreError: 100429:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 2348.961230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.146310] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2355.287710] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2359.202796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.381356] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2359.388257] LustreError: 101335:0:(tgt_mount.c:1527:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2359.392491] LustreError: 101335:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -2 [ 2359.396531] LustreError: 101335:0:(tgt_mount.c:1766:server_put_super()) no obd lustre-MDT0000 [ 2359.503111] Lustre: server umount lustre-MDT0000 complete [ 2359.508568] LustreError: 101335:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2373.090865] LustreError: 101767:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2373.191628] Lustre: server umount lustre-OST0000 complete [ 2377.627471] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 2381.795255] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2386.261818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2389.645944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2392.837827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2395.910981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2399.271879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2399.304172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2399.439047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2399.458131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2399.505062] Lustre: lustre-MDT0000: new disk, initializing [ 2399.565814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2399.577925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2401.754869] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2408.486871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2408.533333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2408.593599] Lustre: 104085:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2408.635731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2408.637816] Lustre: Skipped 1 previous similar message [ 2408.686297] Lustre: lustre-MDT0001: new disk, initializing [ 2408.750371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2408.794661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2408.800368] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2411.175157] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2416.179344] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2418.412098] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2422.473583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2422.539407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2422.685068] Lustre: lustre-OST0000: new disk, initializing [ 2422.687780] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2422.765428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.044210] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2429.481112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2429.489149] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2429.559318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2430.662531] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2432.848482] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2432.947368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2434.985717] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2435.100400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2439.653554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2439.656292] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.662903] Lustre: Skipped 1 previous similar message [ 2439.665456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2439.668236] Lustre: Skipped 1 previous similar message [ 2444.776199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.778746] Lustre: Skipped 1 previous similar message [ 2449.890852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2449.893875] Lustre: Skipped 1 previous similar message [ 2450.399545] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2450.422133] LustreError: 106993:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2450.426815] LustreError: 106993:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2450.511973] Lustre: server umount lustre-OST0000 complete [ 2455.010215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2455.015657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2455.020268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2455.023651] Lustre: Skipped 1 previous similar message [ 2455.031042] Lustre: Skipped 1 previous similar message [ 2458.601608] LustreError: 107195:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2458.604575] LustreError: 107195:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2458.663311] Lustre: server umount lustre-MDT0000 complete [ 2460.764085] LustreError: 104077:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442440 with bad export cookie 12132836802185585548 [ 2460.768134] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.769138] LustreError: 104077:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2460.859565] LustreError: 107396:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2460.862591] LustreError: 107396:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2460.974672] Lustre: server umount lustre-MDT0001 complete [ 2464.792694] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 07:20:42 (1753442442) [ 2466.367613] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2474.053897] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 2478.265631] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2483.724047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2487.801029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2491.184702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2494.570980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2497.876984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2497.920754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2498.039347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2498.061731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2498.105541] Lustre: lustre-MDT0000: new disk, initializing [ 2498.154778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.170276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2500.181215] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2505.678185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2505.720059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2505.771215] Lustre: 110513:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2505.802946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2505.805330] Lustre: Skipped 1 previous similar message [ 2505.845765] Lustre: lustre-MDT0001: new disk, initializing [ 2505.903124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2505.907292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2507.515947] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2511.256553] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2512.905634] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2515.812660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2515.851305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2515.969939] Lustre: lustre-OST0000: new disk, initializing [ 2515.972437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2516.012581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2516.015714] Lustre: Skipped 1 previous similar message [ 2516.482843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2516.487547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2516.538575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2518.471228] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2522.179448] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2523.904404] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2524.000147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2525.796958] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2525.904301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2531.810208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2531.813855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.815155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2531.818220] LustreError: Skipped 1 previous similar message [ 2532.910229] LustreError: 113418:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2532.913726] LustreError: 113418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2532.988382] Lustre: server umount lustre-OST0000 complete [ 2536.929671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.929911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2536.934931] Lustre: Skipped 1 previous similar message [ 2540.867938] Lustre: server umount lustre-MDT0000 complete [ 2542.049438] LustreError: 110525:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.059476] LustreError: 110525:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2542.610231] LustreError: 110505:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442521 with bad export cookie 12132836802185586423 [ 2542.613597] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2542.613752] LustreError: 110505:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2542.696068] LustreError: 113823:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2542.698878] LustreError: 113823:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2542.800679] Lustre: server umount lustre-MDT0001 complete [ 2546.805750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2547.032673] LustreError: 114299:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.079571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.800108] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2552.288776] LustreError: 114299:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.782901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2554.841601] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2557.391873] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2559.007068] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2562.184029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2564.912613] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2567.402991] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2575.711872] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2577.889728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2577.894116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.901076] Lustre: Skipped 1 previous similar message [ 2577.904393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2577.907018] Lustre: Skipped 3 previous similar messages [ 2579.056192] LustreError: 116620:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2579.059265] LustreError: 116620:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2579.136490] Lustre: server umount lustre-OST0000 complete [ 2583.012959] LustreError: 114299:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.079791] Lustre: server umount lustre-MDT0000 complete [ 2584.891701] LustreError: 114278:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442564 with bad export cookie 12132836802185587305 [ 2584.896520] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.896972] LustreError: 114278:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2585.066148] Lustre: server umount lustre-MDT0001 complete [ 2590.870850] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 2592.319605] Key type lgssc unregistered [ 2592.464817] LNet: 117548:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2593.508960] LNet: Removed LNI 192.168.203.149@tcp [ 2593.921824] Key type .llcrypt unregistered [ 2593.923886] Key type ._llcrypt unregistered [ 2601.357835] Key type ._llcrypt registered [ 2601.359229] Key type .llcrypt registered [ 2601.410478] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 2608.806874] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 2609.175364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2609.183876] alg: No test for adler32 (adler32-zlib) [ 2610.060064] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 2610.161739] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 2610.164868] LNet: Accept secure, port 988 [ 2611.751214] Key type lgssc registered [ 2612.253997] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2616.161182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2618.823777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2621.395384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2624.360140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2627.539395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2627.572988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2627.582731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.703984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2628.731273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2628.777455] Lustre: lustre-MDT0000: new disk, initializing [ 2628.837898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2628.846751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2630.562285] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2635.871833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2635.914575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2635.965415] Lustre: 120628:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2635.999861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2636.002701] Lustre: Skipped 1 previous similar message [ 2636.046510] Lustre: lustre-MDT0001: new disk, initializing [ 2636.120630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2636.147037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2636.153782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2637.994630] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2642.080781] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2644.022895] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2647.611819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2647.646748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2647.764613] Lustre: lustre-OST0000: new disk, initializing [ 2647.767430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2647.810298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.233349] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2654.084598] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2655.753553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2655.758512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2655.811967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2655.826465] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2655.925546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2657.588182] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2657.703413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2660.835951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2660.836248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2660.844915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2660.848975] LustreError: Skipped 1 previous similar message [ 2660.853454] Lustre: Skipped 1 previous similar message [ 2664.691644] LustreError: 123533:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2664.785691] Lustre: server umount lustre-OST0000 complete [ 2666.976455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2666.981053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2666.988155] Lustre: Skipped 1 previous similar message [ 2666.991756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2671.072651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2671.073288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.513163] LustreError: 123736:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2672.515561] LustreError: 123736:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2672.552987] Lustre: server umount lustre-MDT0000 complete [ 2674.089108] LustreError: 120622:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442653 with bad export cookie 4328223691527296010 [ 2674.091077] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2674.094878] LustreError: 120622:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2674.172226] LustreError: 123939:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2674.175055] LustreError: 123939:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2674.264915] Lustre: server umount lustre-MDT0001 complete [ 2677.551441] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 07:24:15 (1753442655) [ 2680.211040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.441866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2682.059241] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2685.380448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2685.547653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2687.183240] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2689.359219] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2690.704519] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2695.584595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.585976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.589961] Lustre: Skipped 1 previous similar message [ 2695.592605] Lustre: Skipped 1 previous similar message [ 2697.595715] LustreError: 125941:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2697.598028] LustreError: 125941:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2697.627558] Lustre: server umount lustre-MDT0000 complete [ 2699.133547] LustreError: 124593:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442678 with bad export cookie 4328223691527296878 [ 2699.137954] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2699.139053] LustreError: 124593:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2699.318617] Lustre: server umount lustre-MDT0001 complete [ 2702.620612] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 07:24:40 (1753442680) [ 2705.239456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2722.783615] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2728.927855] LustreError: 126785:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 2757.599355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.765925] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2761.804814] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2762.567969] LustreError: 127443:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2762.570943] LustreError: 127443:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2762.642713] Lustre: server umount lustre-OST0000 complete [ 2765.920318] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 07:25:44 (1753442744) [ 2768.624656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.802236] LustreError: 128113:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.811536] LustreError: 128113:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2770.300395] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2773.875877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2773.986384] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2773.989758] Lustre: Skipped 1 previous similar message [ 2774.071912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2774.075190] Lustre: Skipped 1 previous similar message [ 2775.806491] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2777.994670] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2779.472264] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2782.124245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2784.437720] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2786.572409] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2799.583935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2799.588304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.594951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2805.121122] LustreError: 130361:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2805.123743] LustreError: 130361:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2805.157381] Lustre: server umount lustre-MDT0000 complete [ 2806.240505] LustreError: 128112:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2806.247258] LustreError: 128112:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2806.630789] LustreError: 128092:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442785 with bad export cookie 4328223691527297529 [ 2806.633770] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2806.635825] LustreError: 128092:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2806.792337] Lustre: server umount lustre-MDT0001 complete [ 2818.498695] Lustre: server umount lustre-OST0000 complete [ 2821.789651] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 07:26:40 (1753442800) [ 2824.577569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.799307] LustreError: 131434:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.851680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.854756] Lustre: Skipped 1 previous similar message [ 2826.583699] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2830.090375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2831.814774] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2834.019554] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2835.590079] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2838.264245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2840.617530] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2842.636626] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2844.181100] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2850.725924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2851.448689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 2852.938969] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2853.037410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2855.906118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2855.907502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2855.908975] LustreError: Skipped 1 previous similar message [ 2855.914756] Lustre: Skipped 2 previous similar messages [ 2855.917943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2855.921604] Lustre: Skipped 4 previous similar messages [ 2859.888982] LustreError: 134193:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2859.892214] LustreError: 134193:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2859.965608] Lustre: server umount lustre-OST0000 complete [ 2866.144614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2867.483431] Lustre: server umount lustre-MDT0000 complete [ 2868.831525] LustreError: 134194:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442848 with bad export cookie 4328223691527298537 [ 2868.835367] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2868.836491] LustreError: 134194:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2868.983890] Lustre: server umount lustre-MDT0001 complete [ 2872.210921] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 07:27:30 (1753442850) [ 2874.653159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2891.103557] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2897.251196] LustreError: 135241:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 2927.007316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.010928] Lustre: Skipped 2 previous similar messages [ 2929.637055] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2931.947695] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2934.871427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.045667] LustreError: 135268:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.121546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 2936.627411] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2939.967543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2941.576313] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 2943.733034] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2945.212814] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2946.853763] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2946.966335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2948.599294] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2948.689359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2950.624132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2950.627471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.632767] Lustre: Skipped 2 previous similar messages [ 2950.635434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.638207] Lustre: Skipped 2 previous similar messages [ 2955.582048] LustreError: 137984:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2955.584660] LustreError: 137984:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2955.610273] Lustre: server umount lustre-MDT0000 complete [ 2956.768090] LustreError: 137031:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2956.774699] LustreError: 137031:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2957.001294] LustreError: 135248:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753442936 with bad export cookie 4328223691527299552 [ 2957.004568] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2957.005645] LustreError: 135248:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2957.146592] Lustre: server umount lustre-MDT0001 complete [ 2967.866269] Lustre: server umount lustre-OST0000 complete [ 2971.406498] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 07:29:09 (1753442949) [ 2974.178368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2991.711894] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2997.855452] LustreError: 139032:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 3029.741321] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3032.043844] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3034.960223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.162371] LustreError: 139059:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.222275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3036.783523] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3040.429077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3042.219170] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3044.348460] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3045.761258] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3048.453981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3048.499526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3048.583223] Lustre: lustre-OST0001: new disk, initializing [ 3048.585836] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3049.593830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3049.597766] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3049.633639] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3050.966976] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3054.472940] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3056.090041] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3056.175820] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3057.731202] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3057.838726] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3060.200749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3060.202166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3060.209224] Lustre: Skipped 3 previous similar messages [ 3060.214197] Lustre: Skipped 4 previous similar messages [ 3064.687294] LustreError: 142726:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3064.689394] LustreError: 142726:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3064.716518] Lustre: server umount lustre-OST0000 complete [ 3065.315744] LustreError: 139048:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3065.334042] LustreError: 139048:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3070.436412] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3072.728666] Lustre: server umount lustre-OST0001 complete [ 3080.532547] Lustre: server umount lustre-MDT0000 complete [ 3082.700316] LustreError: 139041:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443061 with bad export cookie 4328223691527300497 [ 3082.703129] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3082.706108] LustreError: 139041:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3082.928936] Lustre: server umount lustre-MDT0001 complete [ 3086.449734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3089.467510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3092.876526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3096.129084] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3102.575473] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 07:31:19 (1753443079) [ 3104.230372] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3105.764946] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3107.185180] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 07:31:24 (1753443084) [ 3112.986405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.183469] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3113.204840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3113.335630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.337972] Lustre: Skipped 6 previous similar messages [ 3117.595652] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3126.076934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3126.155276] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3126.172411] Lustre: 144968:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3129.496773] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3133.605841] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3136.466538] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3141.194287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3141.310644] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3144.589412] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3147.209564] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3148.856346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3149.361084] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3149.474094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3151.590313] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3151.709971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3153.891068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3153.891461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.894901] LustreError: Skipped 1 previous similar message [ 3153.902747] Lustre: Skipped 5 previous similar messages [ 3159.066965] Lustre: server umount lustre-OST0000 complete [ 3175.838238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3178.934039] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3182.118481] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3184.303185] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3194.402911] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3194.410769] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.149@tcp (at 0@lo) [ 3194.410794] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3194.787664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3196.589353] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3196.702467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3198.493847] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff993018415000.ost_server_uuid 50 [ 3199.324128] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff993018415000.ost_server_uuid in FULL state after 0 sec [ 3199.411486] LustreError: 144977:0:(ldlm_lockd.c:728:ldlm_handle_ast_error()) ### client (nid 192.168.203.49@tcp) returned error from blocking AST (req@00000000b611b79f x1838617816399872 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000089c5ef70/0x3c10f083f9ea2ee4 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.49@tcp remote: 0x5bf561de93de3855 expref: 5 pid: 144976 timeout: 3299 lvb_type: 0 [ 3199.427607] LustreError: lustre-MDT0000: A client on nid 192.168.203.49@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3199.431572] LustreError: 144965:0:(ldlm_lockd.c:257:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.49@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000089c5ef70/0x3c10f083f9ea2ee4 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.49@tcp remote: 0x5bf561de93de3855 expref: 6 pid: 144976 timeout: 0 lvb_type: 0 [ 3204.580341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.583941] Lustre: Skipped 7 previous similar messages [ 3207.411247] LustreError: 149538:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3207.424757] LustreError: 149538:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3207.525640] Lustre: server umount lustre-OST0000 complete [ 3215.676509] Lustre: server umount lustre-MDT0000 complete [ 3217.470346] LustreError: 144960:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443196 with bad export cookie 4328223691527301673 [ 3217.472563] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3217.476123] LustreError: 144960:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3223.759213] Lustre: server umount lustre-MDT0001 complete [ 3230.926994] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3232.770855] Key type lgssc unregistered [ 3232.955247] LNet: 150469:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3234.021741] LNet: Removed LNI 192.168.203.149@tcp [ 3234.493168] Key type .llcrypt unregistered [ 3234.496980] Key type ._llcrypt unregistered [ 3246.401775] Key type ._llcrypt registered [ 3246.403367] Key type .llcrypt registered [ 3246.467606] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3247.010116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3247.025801] alg: No test for adler32 (adler32-zlib) [ 3247.940953] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 3248.077452] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3248.081410] LNet: Accept secure, port 988 [ 3249.696075] Key type lgssc registered [ 3250.279421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3254.655271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3254.672809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.987589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.238878] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3263.101590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3263.351663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3265.424649] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3268.395180] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3270.196221] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3273.928829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3274.154382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.824781] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3279.855900] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3281.325160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3294.349963] Lustre: Failing over lustre-MDT0000 [ 3294.483935] LustreError: 154075:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3294.534434] Lustre: server umount lustre-MDT0000 complete [ 3298.785059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.787521] LustreError: 151825:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.793868] Lustre: Skipped 2 previous similar messages [ 3298.808918] LustreError: 151825:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 3303.906659] LustreError: 151824:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.915154] LustreError: 151824:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3306.056143] LustreError: 151810:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.49@tcp arrived at 1753443285 with bad export cookie 6318994781223084594 [ 3306.062435] LustreError: 151810:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3309.027641] LustreError: 151829:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.035279] LustreError: 151829:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3310.069158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.143710] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3310.325417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.352987] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3312.360731] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3313.576752] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3313.583176] Lustre: lustre-MDT0000: Denying connection for new client 2c598f64-29a7-4321-8d06-78d5c151738a (at 192.168.203.49@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 3315.690991] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3324.386759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3324.390584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.398953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3325.921404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3325.924495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.041497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3331.046323] Lustre: Skipped 2 previous similar messages [ 3335.135119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3335.160229] LustreError: 155001:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3335.162211] LustreError: 155001:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3335.266583] Lustre: server umount lustre-OST0000 complete [ 3337.327631] LustreError: 155204:0:(ldlm_lib.c:2916:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3337.332637] Lustre: 154574:0:(ldlm_lib.c:2319:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3337.337457] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3337.351904] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3337.357655] LustreError: 154574:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000de360909 x1838618485236608/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3337.366564] LustreError: 154574:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3337.371579] LustreError: 154574:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3337.376904] Lustre: lustre-MDT0000: Recovery over after 0:24, of 2 clients 0 recovered and 2 were evicted. [ 3337.488426] LustreError: 155204:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3337.491149] LustreError: 155204:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3337.545764] Lustre: server umount lustre-MDT0000 complete [ 3339.261577] LustreError: 151810:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443318 with bad export cookie 6318994781223084930 [ 3339.265611] LustreError: 152535:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.265799] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3339.266680] LustreError: 151810:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3345.421818] LustreError: 155405:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3345.425738] LustreError: 155405:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3345.538321] Lustre: server umount lustre-MDT0001 complete [ 3351.648876] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3353.248954] Key type lgssc unregistered [ 3353.455629] LNet: 155932:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3354.471932] LNet: Removed LNI 192.168.203.149@tcp [ 3354.937900] Key type .llcrypt unregistered [ 3354.940489] Key type ._llcrypt unregistered [ 3367.760478] Key type ._llcrypt registered [ 3367.762017] Key type .llcrypt registered [ 3367.847537] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3368.487927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3368.498269] alg: No test for adler32 (adler32-zlib) [ 3369.533102] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 3369.708415] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3369.711969] LNet: Accept secure, port 988 [ 3371.335281] Key type lgssc registered [ 3372.087370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3377.378072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3377.390736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.666721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.038870] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3386.502199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3386.766881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3389.282886] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3392.456356] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3394.550226] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3398.100165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3398.289655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3401.449748] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3403.432510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3404.144891] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3408.865672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.867360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3408.874473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3409.327181] LustreError: 159521:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3409.401381] Lustre: server umount lustre-OST0000 complete [ 3411.393209] LustreError: 159722:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3411.396526] LustreError: 159722:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3411.464590] Lustre: server umount lustre-MDT0000 complete [ 3412.451310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3412.455357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3412.461855] LustreError: 158001:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3412.477548] LustreError: 158001:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3414.126387] LustreError: 157273:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443393 with bad export cookie 9176248319494617367 [ 3414.133757] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3414.136305] LustreError: 157273:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3420.308270] LustreError: 159924:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3420.326810] LustreError: 159924:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3420.707478] Lustre: server umount lustre-MDT0001 complete [ 3435.238217] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3439.346356] Key type lgssc unregistered [ 3439.796689] LNet: 160452:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3440.871765] LNet: Removed LNI 192.168.203.149@tcp [ 3442.139277] Key type .llcrypt unregistered [ 3442.148683] Key type ._llcrypt unregistered [ 3483.683372] Key type ._llcrypt registered [ 3483.685013] Key type .llcrypt registered [ 3483.921440] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3486.231265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3486.301019] alg: No test for adler32 (adler32-zlib) [ 3487.619895] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 3488.014878] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3488.026681] LNet: Accept secure, port 988 [ 3489.815198] Key type lgssc registered [ 3491.201682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3498.599798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3498.627402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.101730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.870725] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3510.249565] LustreError: 162126:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3510.278157] LustreError: 162126:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3511.367149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3511.644712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3514.677865] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3519.078593] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3522.693481] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3529.387614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3529.813274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3534.783482] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3539.484370] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3545.239260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3550.703472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.715874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3550.717031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3550.737676] Lustre: Skipped 1 previous similar message [ 3554.954193] LustreError: 164355:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3555.153164] Lustre: server umount lustre-OST0000 complete [ 3560.928329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.933697] Lustre: Skipped 1 previous similar message [ 3560.944827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3560.951086] Lustre: Skipped 1 previous similar message [ 3564.595579] LustreError: 164556:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3564.598168] LustreError: 164556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3564.681638] Lustre: server umount lustre-MDT0000 complete [ 3566.048648] LustreError: 164054:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.073080] LustreError: 164054:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3567.973569] LustreError: 162105:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443547 with bad export cookie 3313482119717097413 [ 3567.977580] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3567.985884] LustreError: 162105:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3568.150068] LustreError: 164758:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3568.167343] LustreError: 164758:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3568.440494] Lustre: server umount lustre-MDT0001 complete [ 3578.547350] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3581.264080] Key type lgssc unregistered [ 3581.563817] LNet: 165285:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3582.631163] LNet: Removed LNI 192.168.203.149@tcp [ 3583.465373] Key type .llcrypt unregistered [ 3583.473106] Key type ._llcrypt unregistered [ 3601.730664] Key type ._llcrypt registered [ 3601.732815] Key type .llcrypt registered [ 3601.849218] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3602.740982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3602.812862] alg: No test for adler32 (adler32-zlib) [ 3603.953250] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 3604.179798] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3604.188416] LNet: Accept secure, port 988 [ 3605.903202] Key type lgssc registered [ 3606.992703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3615.177321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3615.191969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3616.442072] Lustre: *** cfs_fail_loc=135, val=0*** [ 3616.503780] LustreError: 166760:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000 failed (-2) [ 3616.510143] LustreError: 166760:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.149@tcp: cfg command failed: rc = -2 [ 3616.514677] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3616.514677] [ 3616.521913] LustreError: MGC192.168.203.149@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3616.530951] LustreError: 166719:0:(tgt_mount.c:1527:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3616.540670] LustreError: 166719:0:(tgt_mount.c:2272:server_fill_super()) Unable to start targets: -2 [ 3616.545893] LustreError: 166719:0:(obd_config.c:832:class_cleanup()) Device 5 not setup [ 3616.551547] LustreError: 166719:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3616.698912] Lustre: server umount lustre-MDT0000 complete [ 3616.707894] LustreError: 166719:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3622.052927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.581208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.759517] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3632.099184] LustreError: 167138:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3632.106976] LustreError: 167138:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3632.914355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3633.229388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3636.964580] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3641.808373] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3645.002705] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3646.607492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.616026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.619017] Lustre: Skipped 1 previous similar message [ 3652.065543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3652.073646] Lustre: Skipped 2 previous similar messages [ 3652.832594] LustreError: 168472:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3652.838737] LustreError: 168472:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3652.933074] Lustre: server umount lustre-MDT0000 complete [ 3655.881286] LustreError: 167854:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443635 with bad export cookie 5363704366126531771 [ 3655.882259] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3655.890632] LustreError: 167854:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3656.056580] LustreError: 168673:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3656.060057] LustreError: 168673:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3656.267861] Lustre: server umount lustre-MDT0001 complete [ 3665.540779] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3668.424176] Key type lgssc unregistered [ 3668.752332] LNet: 169202:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3669.798070] LNet: Removed LNI 192.168.203.149@tcp [ 3670.510599] Key type .llcrypt unregistered [ 3670.512246] Key type ._llcrypt unregistered [ 3690.113266] Key type ._llcrypt registered [ 3690.114499] Key type .llcrypt registered [ 3690.220744] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3690.543635] Key type .llcrypt unregistered [ 3690.545386] Key type ._llcrypt unregistered [ 3705.916311] Key type ._llcrypt registered [ 3705.917863] Key type .llcrypt registered [ 3706.005692] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3706.959510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3707.095781] alg: No test for adler32 (adler32-zlib) [ 3708.107714] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 3708.305892] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3708.308684] LNet: Accept secure, port 988 [ 3709.984087] Key type lgssc registered [ 3710.991866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3717.326328] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3717.343707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3734.943302] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3740.063493] LustreError: 171276:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.149@tcp: failed processing log, type 4: rc = -110 [ 3770.847400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.227760] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3780.984388] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3787.483921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.072466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.151837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3791.536179] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3793.389669] LustreError: 171302:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3793.399640] LustreError: 171302:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3798.497654] LustreError: 171302:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3798.825574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3799.219171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3802.716871] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3808.276207] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3811.794929] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3815.729891] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3829.216049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3829.232916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.268474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3830.240457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3830.256684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3830.274317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3830.319712] Lustre: Skipped 1 previous similar message [ 3832.748069] LustreError: 173934:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3833.007321] Lustre: server umount lustre-OST0000 complete [ 3840.484273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3840.495816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3840.514052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.237799] LustreError: 174135:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3843.241639] LustreError: 174135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3843.341046] Lustre: server umount lustre-MDT0000 complete [ 3845.601060] LustreError: 172199:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3846.839439] LustreError: 171283:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443826 with bad export cookie 15993795559261587190 [ 3846.851664] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3846.861813] LustreError: 171283:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3847.002455] LustreError: 174337:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3847.023225] LustreError: 174337:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3847.296699] Lustre: server umount lustre-MDT0001 complete [ 3857.138624] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 3859.701042] Key type lgssc unregistered [ 3859.946125] LNet: 174865:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3860.969901] LNet: Removed LNI 192.168.203.149@tcp [ 3861.755814] Key type .llcrypt unregistered [ 3861.757346] Key type ._llcrypt unregistered [ 3879.748666] Key type ._llcrypt registered [ 3879.750428] Key type .llcrypt registered [ 3879.842202] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 3880.610642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3880.726825] alg: No test for adler32 (adler32-zlib) [ 3881.769401] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 3881.979048] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 3881.987457] LNet: Accept secure, port 988 [ 3883.703182] Key type lgssc registered [ 3884.585469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3890.968536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3890.991809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.437518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3895.576587] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3901.931560] LustreError: 176226:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.949793] LustreError: 176226:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3903.290228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3903.568172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3907.703584] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3913.057934] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3916.442653] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3922.419277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3922.705162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3928.565952] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3933.378090] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3937.150390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3941.286030] Lustre: Failing over lustre-MDT0000 [ 3941.415805] LustreError: 178504:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3941.499968] Lustre: server umount lustre-MDT0000 complete [ 3942.369057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.377126] LustreError: 177834:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3942.390565] Lustre: Skipped 1 previous similar message [ 3942.410857] LustreError: 177834:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3944.084181] LustreError: 177834:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.490608] LustreError: 176227:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3947.521914] LustreError: 176227:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3952.611039] LustreError: 176227:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.633081] LustreError: 176227:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3958.390441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.469684] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3958.714473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3958.773213] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3959.289038] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3962.054466] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 3963.877351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3963.897699] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3963.959522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3965.546579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3969.024138] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3983.847279] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3983.853333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.863490] Lustre: Skipped 1 previous similar message [ 3983.870109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.355366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3987.577584] LustreError: 179710:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3987.585325] LustreError: 179710:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3987.743612] Lustre: server umount lustre-OST0000 complete [ 3994.592254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.596588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3994.599371] Lustre: Skipped 2 previous similar messages [ 3994.605159] Lustre: Skipped 3 previous similar messages [ 3996.831991] LustreError: 179911:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3996.838336] LustreError: 179911:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3997.011993] Lustre: server umount lustre-MDT0000 complete [ 3999.718704] LustreError: 177211:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3999.749720] LustreError: 177211:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 4000.305767] LustreError: 178506:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753443979 with bad export cookie 7031147428206517223 [ 4000.309257] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4000.325852] LustreError: 178506:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4000.492528] LustreError: 180112:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4000.496615] LustreError: 180112:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4000.662875] Lustre: server umount lustre-MDT0001 complete [ 4010.450103] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 4012.856847] Key type lgssc unregistered [ 4013.116673] LNet: 180639:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4014.180791] LNet: Removed LNI 192.168.203.149@tcp [ 4014.903069] Key type .llcrypt unregistered [ 4014.906588] Key type ._llcrypt unregistered [ 4037.407824] Key type ._llcrypt registered [ 4037.409876] Key type .llcrypt registered [ 4037.505762] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4039.000808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4039.024813] alg: No test for adler32 (adler32-zlib) [ 4040.102513] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 4040.361305] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 4040.367545] LNet: Accept secure, port 988 [ 4042.047176] Key type lgssc registered [ 4043.267231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4050.577340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4050.605243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.932788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4055.057870] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4061.841968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4062.201857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4065.224833] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4069.695140] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4072.086239] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4076.829579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4077.064660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.123508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4081.075679] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4085.246477] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4087.620926] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4095.793553] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4103.191223] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4112.865373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4112.867568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.878885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4112.885013] Lustre: Skipped 1 previous similar message [ 4117.988384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4117.990825] Lustre: Skipped 1 previous similar message [ 4118.327422] LustreError: 184426:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4118.472041] Lustre: server umount lustre-OST0000 complete [ 4123.106885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.112511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4123.116347] Lustre: Skipped 1 previous similar message [ 4127.449127] LustreError: 184627:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4127.454960] LustreError: 184627:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4127.530698] Lustre: server umount lustre-MDT0000 complete [ 4128.224084] LustreError: 183124:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4128.234938] LustreError: 183124:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4130.318958] LustreError: 182003:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444109 with bad export cookie 14984890697604615509 [ 4130.327302] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4130.328143] LustreError: 182003:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4130.461291] LustreError: 184828:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4130.471914] LustreError: 184828:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4130.698628] Lustre: server umount lustre-MDT0001 complete [ 4139.766956] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 4142.163821] Key type lgssc unregistered [ 4142.468665] LNet: 185354:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4143.525610] LNet: Removed LNI 192.168.203.149@tcp [ 4144.186882] Key type .llcrypt unregistered [ 4144.188516] Key type ._llcrypt unregistered [ 4162.687887] Key type ._llcrypt registered [ 4162.689487] Key type .llcrypt registered [ 4162.808881] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4163.482175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4163.558056] alg: No test for adler32 (adler32-zlib) [ 4164.602522] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 4164.808907] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 4164.812706] LNet: Accept secure, port 988 [ 4166.455217] Key type lgssc registered [ 4167.254772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4173.777221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4173.802813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4175.221317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4178.292130] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4185.573597] LustreError: 186716:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4185.585208] LustreError: 186716:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4185.883206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4186.232178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4188.915491] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4192.549715] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4195.140648] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4200.039357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4200.304084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4204.200662] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4208.330784] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4220.793388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4228.069195] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4240.642418] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4251.044645] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4256.096783] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4266.980096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.980697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4266.986522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4266.987202] Lustre: Skipped 1 previous similar message [ 4272.099939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4272.105419] Lustre: Skipped 2 previous similar messages [ 4277.226941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4277.234292] Lustre: Skipped 1 previous similar message [ 4281.313808] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4281.364204] LustreError: 190602:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4281.553855] Lustre: server umount lustre-OST0000 complete [ 4287.456753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4287.470544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4287.477139] Lustre: Skipped 1 previous similar message [ 4292.578293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4292.593663] Lustre: Skipped 1 previous similar message [ 4299.231470] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4299.345534] LustreError: 190805:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4299.348579] LustreError: 190805:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4299.485902] Lustre: server umount lustre-MDT0000 complete [ 4302.819464] LustreError: 186711:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.829854] LustreError: 186711:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4303.056760] LustreError: 189565:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444282 with bad export cookie 14149022331911853004 [ 4303.066056] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4303.071790] LustreError: 189565:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4303.183392] LustreError: 191007:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4303.207975] LustreError: 191007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4303.399419] Lustre: server umount lustre-MDT0001 complete [ 4313.653845] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 4316.365282] Key type lgssc unregistered [ 4316.618536] LNet: 191533:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4317.671900] LNet: Removed LNI 192.168.203.149@tcp [ 4318.422295] Key type .llcrypt unregistered [ 4318.427990] Key type ._llcrypt unregistered [ 4336.296195] Key type ._llcrypt registered [ 4336.297561] Key type .llcrypt registered [ 4336.361488] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4337.128143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4337.161585] alg: No test for adler32 (adler32-zlib) [ 4338.196544] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 4338.369682] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 4338.374533] LNet: Accept secure, port 988 [ 4340.032736] Key type lgssc registered [ 4340.896827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4346.964316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4346.983492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.467488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.574967] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4358.030528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4358.381704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4361.002619] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4364.487352] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4366.766079] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4372.074809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4372.355360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4374.403580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4376.507957] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4380.629792] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4393.957313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4393.961089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4393.965165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4393.974169] Lustre: Skipped 1 previous similar message [ 4396.344203] LustreError: 195272:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4396.490386] Lustre: server umount lustre-OST0000 complete [ 4399.588105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4399.592737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4399.606353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4399.608934] Lustre: Skipped 1 previous similar message [ 4404.193953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4404.197794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4405.513145] LustreError: 195473:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4405.516666] LustreError: 195473:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4405.585254] Lustre: server umount lustre-MDT0000 complete [ 4408.341382] LustreError: 192876:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444387 with bad export cookie 17168117527185096939 [ 4408.346718] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4408.350858] LustreError: 192876:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4408.458160] LustreError: 195675:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4408.461618] LustreError: 195675:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4408.638403] Lustre: server umount lustre-MDT0001 complete [ 4416.532778] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 4418.669599] Key type lgssc unregistered [ 4418.952377] LNet: 196203:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4420.010569] LNet: Removed LNI 192.168.203.149@tcp [ 4420.670656] Key type .llcrypt unregistered [ 4420.673365] Key type ._llcrypt unregistered [ 4435.609775] Key type ._llcrypt registered [ 4435.611544] Key type .llcrypt registered [ 4435.709746] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4436.372185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4436.475293] alg: No test for adler32 (adler32-zlib) [ 4437.542670] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 4437.761254] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 4437.765973] LNet: Accept secure, port 988 [ 4439.440138] Key type lgssc registered [ 4440.456354] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4447.336873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4447.363391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4448.666378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.495742] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4457.954041] LustreError: 197418:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.961346] LustreError: 197418:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4458.420841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4458.685670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4461.405220] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4465.450762] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4468.232727] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4473.152092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4473.316081] LustreError: 199003:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4473.416916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.677990] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4478.442344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4479.731853] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4488.673203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4488.676949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.683181] LustreError: Skipped 1 previous similar message [ 4488.692248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4488.703626] Lustre: Skipped 1 previous similar message [ 4490.173025] LustreError: 199795:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4490.354197] Lustre: server umount lustre-OST0000 complete [ 4493.796834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4493.795475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4493.806087] Lustre: Skipped 2 previous similar messages [ 4498.913272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4498.919210] Lustre: Skipped 2 previous similar messages [ 4499.017681] LustreError: 199997:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4499.024105] LustreError: 199997:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4499.108283] Lustre: server umount lustre-MDT0000 complete [ 4501.603729] LustreError: 197399:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444480 with bad export cookie 989561996023240923 [ 4501.604187] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4501.617253] LustreError: 197399:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4501.715589] LustreError: 200198:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4501.722201] LustreError: 200198:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4501.880388] Lustre: server umount lustre-MDT0001 complete [ 4510.492751] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 4513.214745] Key type lgssc unregistered [ 4513.475316] LNet: 200725:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4514.544215] LNet: Removed LNI 192.168.203.149@tcp [ 4515.186904] Key type .llcrypt unregistered [ 4515.188505] Key type ._llcrypt unregistered [ 4529.203429] Key type ._llcrypt registered [ 4529.204887] Key type .llcrypt registered [ 4529.271803] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4529.864258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4529.902374] alg: No test for adler32 (adler32-zlib) [ 4530.841928] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 4530.973642] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 4530.977844] LNet: Accept secure, port 988 [ 4532.608460] Key type lgssc registered [ 4533.368966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4538.670633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4538.688134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.979979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4542.292587] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4547.703080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4547.943807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4550.875459] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4554.638911] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4557.345312] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4562.056660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4562.342421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4565.435371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4566.387308] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4570.082625] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4577.105542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4577.146620] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4577.318034] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4581.612617] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4585.748832] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4587.156923] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4598.106636] Lustre: Permanently deactivating lustre-OST0001 [ 4598.117822] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4608.029215] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4614.895813] Lustre: Permanently reactivating lustre-OST0001 [ 4614.907696] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4614.911367] Lustre: Skipped 2 previous similar messages [ 4615.479948] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.487617] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4615.494839] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4615.502904] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 192.168.203.149@tcp (at 0@lo) [ 4615.528736] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4629.107794] LustreError: 205433:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4629.153826] Lustre: server umount lustre-OST0001 complete [ 4631.014195] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4631.015942] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4631.026113] Lustre: Skipped 1 previous similar message [ 4631.032413] LustreError: 203528:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.046093] LustreError: 203528:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4632.034564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4632.038665] LustreError: Skipped 1 previous similar message [ 4632.041522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.055976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4635.105104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4635.110299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.119787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4636.132851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4636.135044] LustreError: 204194:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4636.138662] Lustre: Skipped 1 previous similar message [ 4637.748125] LustreError: 205633:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4637.750979] LustreError: 205633:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4637.856586] Lustre: server umount lustre-OST0000 complete [ 4640.226370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4640.230620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4640.246650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4646.231896] LustreError: 205834:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4646.235695] LustreError: 205834:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4646.274636] Lustre: server umount lustre-MDT0000 complete [ 4646.372258] LustreError: 201936:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.387090] LustreError: 201936:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4648.373796] LustreError: 201920:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444627 with bad export cookie 17932578604809572439 [ 4648.377220] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.386835] LustreError: 201920:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4648.488226] LustreError: 206037:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4648.490591] LustreError: 206037:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4648.668280] Lustre: server umount lustre-MDT0001 complete [ 4656.968475] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 4659.157916] Key type lgssc unregistered [ 4659.419470] LNet: 206563:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4660.454883] LNet: Removed LNI 192.168.203.149@tcp [ 4661.027193] Key type .llcrypt unregistered [ 4661.029325] Key type ._llcrypt unregistered [ 4683.855820] Key type ._llcrypt registered [ 4683.857514] Key type .llcrypt registered [ 4683.938958] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 4694.416530] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 4695.161189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4695.168528] alg: No test for adler32 (adler32-zlib) [ 4696.142742] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 4696.308090] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 4696.314047] LNet: Accept secure, port 988 [ 4697.959245] Key type lgssc registered [ 4698.917537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4706.589313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4711.528145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4716.353030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4720.758962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4725.124768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4725.162267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4725.186465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.331187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4726.350619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4726.399913] Lustre: lustre-MDT0000: new disk, initializing [ 4726.455085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.468150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4728.427488] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4735.499479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4735.540420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4735.590752] Lustre: 210472:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4735.638908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4735.642358] Lustre: Skipped 1 previous similar message [ 4735.686700] Lustre: lustre-MDT0001: new disk, initializing [ 4735.748974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4735.794994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4735.806428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4738.326230] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4743.588198] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4745.748028] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4749.986562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4750.040785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4750.203920] Lustre: lustre-OST0000: new disk, initializing [ 4750.207978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4750.268543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.450854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4751.454733] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4751.551527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4753.642265] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4758.636209] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4761.009183] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4761.130169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4763.408435] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4763.575449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4767.203459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4767.209522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.210024] LustreError: Skipped 1 previous similar message [ 4767.212046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4767.218024] Lustre: Skipped 1 previous similar message [ 4771.056115] LustreError: 213377:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4771.174896] Lustre: server umount lustre-OST0000 complete [ 4776.928522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4776.937694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.945877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4776.956754] Lustre: Skipped 1 previous similar message [ 4782.562961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4782.566476] Lustre: Skipped 3 previous similar messages [ 4787.681743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4787.685914] Lustre: Skipped 1 previous similar message [ 4788.191175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4788.238428] LustreError: 213578:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4788.242328] LustreError: 213578:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4788.289933] Lustre: server umount lustre-MDT0000 complete [ 4790.519741] LustreError: 210464:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444769 with bad export cookie 14339842724735980615 [ 4790.520501] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4790.527677] LustreError: 210464:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4790.596653] LustreError: 213781:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4790.599036] LustreError: 213781:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4790.747706] Lustre: server umount lustre-MDT0001 complete [ 4795.087335] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 07:59:33 (1753444773) [ 4798.984635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4799.325578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4801.926359] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4804.583052] LustreError: 214451:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4804.598045] LustreError: 214451:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4808.235144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4808.475129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4810.964719] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4814.521192] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4816.990504] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4821.083704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4821.278128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4824.836473] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 4828.387440] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4830.643980] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4839.847306] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4849.115630] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4854.098168] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4868.225813] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4868.232819] Lustre: Skipped 1 previous similar message [ 4894.763505] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4894.769770] Lustre: Skipped 2 previous similar messages [ 4931.427860] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4931.436902] Lustre: Skipped 4 previous similar messages [ 4997.537356] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 4997.541432] Lustre: Skipped 7 previous similar messages [ 5002.720364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5002.726621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.734653] Lustre: Skipped 1 previous similar message [ 5002.739359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.803082] LustreError: 217937:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5002.806552] LustreError: 217937:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5002.913608] Lustre: server umount lustre-OST0000 complete [ 5005.794426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.800372] Lustre: Skipped 1 previous similar message [ 5010.915155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5010.921625] Lustre: Skipped 3 previous similar messages [ 5015.016945] LustreError: 218142:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5015.024360] LustreError: 218142:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5015.071387] Lustre: server umount lustre-MDT0000 complete [ 5016.034340] LustreError: 215237:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5016.050034] LustreError: 215237:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5017.572396] LustreError: 216967:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753444996 with bad export cookie 14339842724735981504 [ 5017.575764] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5017.578074] LustreError: 216967:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5017.899579] Lustre: server umount lustre-MDT0001 complete [ 5026.161553] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5028.556884] Key type lgssc unregistered [ 5028.810543] LNet: 218871:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5029.861510] LNet: Removed LNI 192.168.203.149@tcp [ 5030.541107] Key type .llcrypt unregistered [ 5030.542745] Key type ._llcrypt unregistered [ 5046.151351] Key type ._llcrypt registered [ 5046.153064] Key type .llcrypt registered [ 5046.218581] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5046.818114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5046.870187] alg: No test for adler32 (adler32-zlib) [ 5047.864087] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 5048.016082] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5048.021181] LNet: Accept secure, port 988 [ 5049.663225] Key type lgssc registered [ 5050.547235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5056.136796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5056.158218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.507156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.148280] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5066.138876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5066.403533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5069.504218] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5073.463646] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5076.227222] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5081.153864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5081.447568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5085.250570] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_default_debug -1 all [ 5088.854336] Lustre: DEBUG MARKER: oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5102.051781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5102.053206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5102.056731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5102.060938] Lustre: Skipped 1 previous similar message [ 5106.485283] LustreError: 222622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5106.568094] Lustre: server umount lustre-OST0000 complete [ 5112.288664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5112.289936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5112.300879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5112.310395] Lustre: Skipped 2 previous similar messages [ 5113.313049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5113.316108] Lustre: Skipped 1 previous similar message [ 5114.592744] LustreError: 222824:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5114.597323] LustreError: 222824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5114.646571] Lustre: server umount lustre-MDT0000 complete [ 5116.888052] LustreError: 222453:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1753445096 with bad export cookie 17096262814076025551 [ 5116.894133] LustreError: MGC192.168.203.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5116.894511] LustreError: 222453:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5116.982381] LustreError: 223025:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5116.986167] LustreError: 223025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5117.106423] Lustre: server umount lustre-MDT0001 complete [ 5123.418430] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5124.917535] Key type lgssc unregistered [ 5125.083926] LNet: 223551:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5126.118695] LNet: Removed LNI 192.168.203.149@tcp [ 5126.617848] Key type .llcrypt unregistered [ 5126.619588] Key type ._llcrypt unregistered [ 5138.834904] Key type ._llcrypt registered [ 5138.837768] Key type .llcrypt registered [ 5138.926457] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5139.116694] Key type .llcrypt unregistered [ 5139.118347] Key type ._llcrypt unregistered [ 5155.102130] Key type ._llcrypt registered [ 5155.103990] Key type .llcrypt registered [ 5155.173540] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing set_hostid [ 5163.800958] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing load_modules_local [ 5164.212343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5164.263650] alg: No test for adler32 (adler32-zlib) [ 5165.157385] Lustre: Lustre: Build Version: 2.16.56_57_g82204de [ 5165.270702] LNet: Added LNI 192.168.203.149@tcp [8/256/0/180] [ 5165.277389] LNet: Accept secure, port 988 [ 5166.896405] Key type lgssc registered [ 5167.535681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5173.763168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5179.086574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5183.310470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5188.036725] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5189.172222] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4930 sec ========== 08:06:06 (1753445166) [ 5190.319163] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:06:08 (1753445168) === [ 5192.492801] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:06:10 (1753445170) === [ 5212.083278] Lustre: DEBUG MARKER: oleg349-server.virtnet: executing unload_modules_local [ 5213.531750] Key type lgssc unregistered [ 5213.693706] LNet: 229621:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5213.703742] LNet: Removed LNI 192.168.203.149@tcp [ 5214.105577] Key type .llcrypt unregistered [ 5214.107301] Key type ._llcrypt unregistered