[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 514701269 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002416] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007479] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011046] Yama: becoming mindful. [ 0.012236] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.022387] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028581] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029210] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030151] *** VALIDATE tmpfs *** [ 0.031441] *** VALIDATE proc *** [ 0.032327] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034312] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036089] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038039] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.043083] debug: unmapping init [mem 0xffffffff8c059000-0xffffffff8c060fff] [ 0.045925] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046705] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047028] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049019] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.055249] rcu: Hierarchical SRCU implementation. [ 0.057521] smp: Bringing up secondary CPUs ... [ 0.058655] x86: Booting SMP configuration: [ 0.059025] .... node #0, CPUs: #1 #2 #3 [ 0.063014] smp: Brought up 1 node, 4 CPUs [ 0.065020] smpboot: Max logical packages: 1 [ 0.066013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146400] node 0 deferred pages initialised in 78ms [ 0.149578] devtmpfs: initialized [ 0.150245] x86/mm: Memory block size: 128MB [ 0.152817] gcov: version magic: 0x41383552 [ 0.153764] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157281] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160334] pinctrl core: initialized pinctrl subsystem [ 0.161000] [ 0.161000] ************************************************************* [ 0.162010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165011] ** ** [ 0.167011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169011] ** ** [ 0.171013] ** This means that this kernel is built to expose internal ** [ 0.173051] ** IOMMU data structures, which may compromise security on ** [ 0.176013] ** your system. ** [ 0.178010] ** ** [ 0.180013] ** If you see this message and you are not debugging the ** [ 0.182013] ** kernel, report this immediately to your vendor! ** [ 0.184010] ** ** [ 0.186013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188013] ************************************************************* [ 0.191654] NET: Registered protocol family 16 [ 0.193435] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201456] cpuidle: using governor menu [ 0.203035] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205536] PCI: Using configuration type 1 for base access [ 0.208134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223162] cryptd: max_cpu_qlen set to 1000 [ 0.225695] ACPI: Added _OSI(Module Device) [ 0.226000] ACPI: Added _OSI(Processor Device) [ 0.229027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.234479] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241357] ACPI: Interpreter enabled [ 0.243072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244011] ACPI: Using IOAPIC for interrupt routing [ 0.246132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249361] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259596] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.262040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.264017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272000] acpiphp: Slot [2] registered [ 0.273200] acpiphp: Slot [3] registered [ 0.275124] acpiphp: Slot [4] registered [ 0.277105] acpiphp: Slot [5] registered [ 0.278183] acpiphp: Slot [6] registered [ 0.280113] acpiphp: Slot [7] registered [ 0.281122] acpiphp: Slot [8] registered [ 0.283116] acpiphp: Slot [9] registered [ 0.284119] acpiphp: Slot [10] registered [ 0.285108] acpiphp: Slot [11] registered [ 0.287103] acpiphp: Slot [12] registered [ 0.289098] acpiphp: Slot [13] registered [ 0.291143] acpiphp: Slot [14] registered [ 0.293115] acpiphp: Slot [15] registered [ 0.294097] acpiphp: Slot [16] registered [ 0.296121] acpiphp: Slot [17] registered [ 0.297103] acpiphp: Slot [18] registered [ 0.299096] acpiphp: Slot [19] registered [ 0.300109] acpiphp: Slot [20] registered [ 0.302097] acpiphp: Slot [21] registered [ 0.303111] acpiphp: Slot [22] registered [ 0.304096] acpiphp: Slot [23] registered [ 0.306161] acpiphp: Slot [24] registered [ 0.308107] acpiphp: Slot [25] registered [ 0.309097] acpiphp: Slot [26] registered [ 0.311094] acpiphp: Slot [27] registered [ 0.312109] acpiphp: Slot [28] registered [ 0.313105] acpiphp: Slot [29] registered [ 0.315127] acpiphp: Slot [30] registered [ 0.316135] acpiphp: Slot [31] registered [ 0.318056] PCI host bridge to bus 0000:00 [ 0.319016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.321023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.326026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.329025] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.331034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.333179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.339577] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346862] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.357781] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.363642] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375244] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377798] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.381048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.383584] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.389016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.403019] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.407764] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.412521] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446017] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.461531] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.468020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.477023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.494015] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.504434] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.512015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.518016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.532025] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.542530] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.547017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.551016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.567019] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.577532] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.588022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.596016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.612019] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.625074] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.632017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.639016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.656020] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.668610] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.671390] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.674371] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.676385] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.679252] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.683122] iommu: Default domain type: Passthrough [ 0.684505] SCSI subsystem initialized [ 0.686147] ACPI: bus type USB registered [ 0.687096] usbcore: registered new interface driver usbfs [ 0.689332] usbcore: registered new interface driver hub [ 0.691073] usbcore: registered new device driver usb [ 0.693175] pps_core: LinuxPPS API ver. 1 registered [ 0.694009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.697078] PTP clock support registered [ 0.700089] EDAC MC: Ver: 3.0.0 [ 0.702140] PCI: Using ACPI for IRQ routing [ 0.704887] NetLabel: Initializing [ 0.706013] NetLabel: domain hash size = 128 [ 0.707010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.710081] NetLabel: unlabeled traffic allowed by default [ 0.712125] vgaarb: loaded [ 0.714256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.716016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.726696] clocksource: Switched to clocksource kvm-clock [ 0.839806] VFS: Disk quotas dquot_6.6.0 [ 0.841442] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.843604] *** VALIDATE ramfs *** [ 0.844779] *** VALIDATE hugetlbfs *** [ 0.846896] pnp: PnP ACPI init [ 0.849104] pnp: PnP ACPI: found 6 devices [ 0.867983] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.871734] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.873616] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.876272] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.878725] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.881489] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.884836] NET: Registered protocol family 2 [ 0.887499] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.892840] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.897946] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.905301] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.909515] TCP: Hash tables configured (established 65536 bind 65536) [ 0.912441] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.915547] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.918455] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.921382] NET: Registered protocol family 1 [ 0.923772] RPC: Registered named UNIX socket transport module. [ 0.926103] RPC: Registered udp transport module. [ 0.927857] RPC: Registered tcp transport module. [ 0.929575] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.932235] NET: Registered protocol family 44 [ 0.933853] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.935763] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.937908] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.939821] PCI: CLS 0 bytes, default 64 [ 0.941606] Unpacking initramfs... [ 2.348172] debug: unmapping init [mem 0xffff8cf07cc54000-0xffff8cf07ffbffff] [ 2.352051] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.354250] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.356591] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.873485] Initialise system trusted keyrings [ 2.874943] Key type blacklist registered [ 2.880183] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.892513] zbud: loaded [ 2.896758] *** VALIDATE nfs *** [ 2.898373] *** VALIDATE nfs4 *** [ 2.900540] pstore: using deflate compression [ 2.904412] Platform Keyring initialized [ 3.007549] NET: Registered protocol family 38 [ 3.009127] Key type asymmetric registered [ 3.010545] Asymmetric key parser 'x509' registered [ 3.012368] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.015325] io scheduler mq-deadline registered [ 3.017164] io scheduler kyber registered [ 3.019329] io scheduler bfq registered [ 3.021953] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.024653] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.028462] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.031596] ACPI: Power Button [PWRF] [ 3.124892] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.218807] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.407855] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.500401] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.717791] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.745277] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.776344] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.782131] Non-volatile memory driver v1.3 [ 3.783728] Linux agpgart interface v0.103 [ 3.817068] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.820715] vda: detected capacity change from 0 to 68530176 [ 3.839266] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.842740] vdb: detected capacity change from 0 to 1073741824 [ 3.862492] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.866592] vdc: detected capacity change from 0 to 2621440000 [ 3.880672] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.883864] vdd: detected capacity change from 0 to 2621440000 [ 3.897945] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.900618] vde: detected capacity change from 0 to 4294967296 [ 3.916740] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.919882] vdf: detected capacity change from 0 to 4294967296 [ 3.925905] libphy: Fixed MDIO Bus: probed [ 3.930389] usbcore: registered new interface driver usbserial_generic [ 3.933156] usbserial: USB Serial support registered for generic [ 3.935949] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.940197] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.942218] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.944285] mousedev: PS/2 mouse device common for all mice [ 3.948823] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.950113] rtc_cmos 00:05: RTC can wake from S4 [ 3.956347] rtc_cmos 00:05: registered as rtc0 [ 3.956502] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.958931] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.965582] intel_pstate: CPU model not supported [ 3.969531] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.971783] hid: raw HID events driver (C) Jiri Kosina [ 3.975405] usbcore: registered new interface driver usbhid [ 3.977336] usbhid: USB HID core driver [ 3.978629] drop_monitor: Initializing network drop monitor service [ 3.981201] Initializing XFRM netlink socket [ 3.982981] NET: Registered protocol family 10 [ 3.986070] Segment Routing with IPv6 [ 3.987581] NET: Registered protocol family 17 [ 3.991433] mpls_gso: MPLS GSO support [ 3.997384] RAS: Correctable Errors collector initialized. [ 3.999679] AVX version of gcm_enc/dec engaged. [ 4.001340] AES CTR mode by8 optimization enabled [ 4.108412] sched_clock: Marking stable (4108383062, 0)->(5007901995, -899518933) [ 4.111586] registered taskstats version 1 [ 4.113676] Loading compiled-in X.509 certificates [ 4.116160] zswap: loaded using pool lzo/zbud [ 4.147308] Key type big_key registered [ 4.158865] Key type encrypted registered [ 4.159991] ima: No TPM chip found, activating TPM-bypass! [ 4.161232] ima: Allocated hash algorithm: sha1 [ 4.162261] ima: No architecture policies found [ 4.163539] evm: Initialising EVM extended attributes: [ 4.164721] evm: security.selinux [ 4.165503] evm: security.ima [ 4.166196] evm: security.capability [ 4.166990] evm: HMAC attrs: 0x1 [ 4.168729] rtc_cmos 00:05: setting system clock to 2025-11-16 23:54:52 UTC (1763337292) [ 4.174152] debug: unmapping init [mem 0xffffffff8d003000-0xffffffff8d1fffff] [ 4.178808] debug: unmapping init [mem 0xffffffff8bd82000-0xffffffff8c058fff] [ 4.187078] Write protecting the kernel read-only data: 28672k [ 4.190791] debug: unmapping init [mem 0xffffffff8a403000-0xffffffff8a5fffff] [ 4.193237] debug: unmapping init [mem 0xffffffff8ad14000-0xffffffff8adfffff] [ 4.227789] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.238060] systemd[1]: Detected virtualization kvm. [ 4.240274] systemd[1]: Detected architecture x86-64. [ 4.242450] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.271052] systemd[1]: No hostname configured. [ 4.272707] systemd[1]: Set hostname to . [ 4.274692] random: systemd: uninitialized urandom read (16 bytes read) [ 4.277185] systemd[1]: Initializing machine ID from random generator. [ 4.316698] random: ln: uninitialized urandom read (6 bytes read) [ 4.422246] random: systemd: uninitialized urandom read (16 bytes read) [ 4.426344] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.434680] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 4.441485] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.065323] device-mapper: uevent: version 1.0.3 [ 5.067621] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.752200] random: fast init done [ 5.757731] virtio_net virtio0 ens2: renamed from eth0 [ 5.815820] scsi host0: ata_piix [ 5.847767] scsi host1: ata_piix [ 5.851501] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.853859] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.215613] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.671379] random: crng init done [ 10.672808] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.113422] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.277591] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.560303] SELinux: Disabled at runtime. [ 12.624105] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.632824] systemd[1]: Detected virtualization kvm. [ 12.634611] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.160360] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.164302] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.169705] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.173628] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.177223] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.184631] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.188508] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ 13.270627] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.692293] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.998123] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.067250] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.189952] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.215972] EDAC sbridge: Ver: 1.1.2 [ 16.143129] Key type dns_resolver registered [ 16.451170] NFS: Registering the id_resolver key type [ 16.453180] Key type id_resolver registered [ 16.454445] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg232-server login: [ 43.281364] libcfs: loading out-of-tree module taints kernel. [ 43.296346] Key type ._llcrypt registered [ 43.297700] Key type .llcrypt registered [ 43.353402] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 55.206968] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 56.393645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 56.402113] alg: No test for adler32 (adler32-zlib) [ 57.683934] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 58.233928] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 59.987071] Key type lgssc registered [ 61.645875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.543450] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 77.208287] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.576373] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 90.128191] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.761668] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.959130] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 111.623599] hrtimer: interrupt took 2645301 ns [ 119.659312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.696227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 119.712834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 120.963781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 120.990849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 121.061579] Lustre: lustre-MDT0000: new disk, initializing [ 121.139068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.151256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 124.416937] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.635490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 135.698267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 135.763225] Lustre: 6485:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 135.798857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 135.803142] Lustre: Skipped 1 previous similar message [ 135.858782] Lustre: lustre-MDT0001: new disk, initializing [ 135.930316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 135.953066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 135.971078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 139.056055] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.468500] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 152.510629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 152.581749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 152.772328] Lustre: lustre-OST0000: new disk, initializing [ 152.784552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 152.860144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.393771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 154.411737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 154.497492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 157.628176] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.525200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.621025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 171.765599] Lustre: lustre-OST0001: new disk, initializing [ 171.771639] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 171.905440] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.454702] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.332496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 182.347513] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 182.437269] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 188.533385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 198.207634] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 208.208992] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing check_logdir /tmp/testlogs/ [ 211.425492] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing yml_node [ 215.685675] Lustre: DEBUG MARKER: Client: 2.16.61.50 [ 217.685876] Lustre: DEBUG MARKER: MDS: 2.16.61.50 [ 219.747451] Lustre: DEBUG MARKER: OSS: 2.16.61.50 [ 221.422863] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 18:58:27 EST 2025 [ 236.485848] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 237.676381] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 253.924232] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 253.925082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 253.928188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 253.950271] Lustre: Skipped 3 previous similar messages [ 253.952514] Lustre: Skipped 2 previous similar messages [ 259.042651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 259.054023] Lustre: Skipped 4 previous similar messages [ 264.166755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 264.184020] Lustre: Skipped 3 previous similar messages [ 266.208614] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 266.340086] LustreError: 12190:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 266.439345] Lustre: server umount lustre-MDT0000 complete [ 269.830962] LustreError: 6478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337558 with bad export cookie 15508523489373311341 [ 269.836536] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 274.400868] LustreError: 11118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 274.400874] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 274.408949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 274.414808] LustreError: 11118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 274.420331] Lustre: Skipped 1 previous similar message [ 276.099328] LustreError: 12393:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 276.106031] LustreError: 12393:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 276.284705] Lustre: server umount lustre-MDT0001 complete [ 279.925504] LustreError: 12595:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 279.929382] LustreError: 12595:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 279.989102] Lustre: server umount lustre-OST0000 complete [ 283.540093] LustreError: 12795:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 283.542658] LustreError: 12795:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 283.740503] Lustre: server umount lustre-OST0001 complete [ 289.369288] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 298.417884] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 308.055254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 314.279400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 320.081810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 325.622452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.017560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.076917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.238469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 332.277861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 332.343875] Lustre: lustre-MDT0000: new disk, initializing [ 332.442406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.471839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 335.984497] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 345.129173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 345.179422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 345.295940] Lustre: 14978:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 345.362083] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 345.366836] Lustre: Skipped 1 previous similar message [ 345.433525] Lustre: lustre-MDT0001: new disk, initializing [ 345.519640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 345.546893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 345.558567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 349.627913] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 356.067736] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 359.571991] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 366.205728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 366.294955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 366.655851] Lustre: lustre-OST0000: new disk, initializing [ 366.673987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 366.795966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.653058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 368.668018] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 368.835570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 372.343693] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 379.288640] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 383.851991] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 384.088631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 388.429628] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 388.696079] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 394.732817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.733423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 394.760923] Lustre: Skipped 1 previous similar message [ 394.770476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 396.484135] LustreError: 17933:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 396.499994] LustreError: 17933:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 396.690078] Lustre: server umount lustre-OST0000 complete [ 401.898645] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 401.901912] LustreError: Skipped 1 previous similar message [ 401.903994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.962876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 404.973656] Lustre: Skipped 3 previous similar messages [ 406.456381] LustreError: 18135:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 406.565087] LustreError: 18135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 406.755478] Lustre: server umount lustre-MDT0000 complete [ 410.091042] LustreError: 14990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 410.119297] LustreError: 14990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 410.430480] LustreError: 14972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337698 with bad export cookie 15508523489373312874 [ 410.438860] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 410.468681] LustreError: 14972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 410.912224] Lustre: server umount lustre-MDT0001 complete [ 414.085707] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:01:40 (1763337700) [ 420.136694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.593586] LustreError: 18860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 420.677849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.243316] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 425.955353] LustreError: 18860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.051532] LustreError: 18860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.239834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 436.579759] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 441.790791] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 444.906852] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 451.639369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 456.975706] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 461.908300] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 475.620115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.620869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.622987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.625702] Lustre: Skipped 2 previous similar messages [ 479.060701] LustreError: 21107:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 479.065522] LustreError: 21107:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 479.206811] Lustre: server umount lustre-OST0000 complete [ 483.810193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 483.817389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.907219] Lustre: server umount lustre-MDT0000 complete [ 490.977405] LustreError: 18860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.989215] LustreError: 18860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 492.254386] LustreError: 18840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337780 with bad export cookie 15508523489373313756 [ 492.262357] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 492.265574] LustreError: 18840:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 492.646185] Lustre: server umount lustre-MDT0001 complete [ 501.254976] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 503.888730] Key type lgssc unregistered [ 504.127921] LNet: 22042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.140219] LNetError: 22042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 504.157447] LNet: Removed LNI 192.168.202.132@tcp [ 505.047204] Key type .llcrypt unregistered [ 505.050250] Key type ._llcrypt unregistered [ 522.919320] Key type ._llcrypt registered [ 522.920784] Key type .llcrypt registered [ 523.005690] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 523.584887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 523.790019] alg: No test for adler32 (adler32-zlib) [ 524.833191] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 525.143887] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 526.832204] Key type lgssc registered [ 527.853028] Lustre: Echo OBD driver; http://www.lustre.org/ [ 534.592709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 534.646128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.159628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.181603] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 546.277155] LustreError: 23416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.292357] LustreError: 23416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 546.528575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 546.902298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 549.917870] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 554.242625] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 557.602734] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 563.242666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 563.555089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.317471] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 573.985862] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 581.120763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 587.237518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 587.248342] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.289254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 591.331657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 591.339432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 591.351291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.361286] Lustre: Skipped 1 previous similar message [ 593.289838] LustreError: 26103:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 593.433576] Lustre: server umount lustre-OST0000 complete [ 598.000402] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 598.003492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.017941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.574054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.575403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.606125] Lustre: Skipped 1 previous similar message [ 603.496973] LustreError: 26305:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 603.524412] LustreError: 26305:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 603.662098] Lustre: server umount lustre-MDT0000 complete [ 606.694204] LustreError: 23416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 606.709958] LustreError: 23416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 607.310674] LustreError: 23396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763337895 with bad export cookie 10476974483563801614 [ 607.321679] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 607.323098] LustreError: 23396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 607.474574] LustreError: 26507:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 607.477678] LustreError: 26507:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 607.663412] Lustre: server umount lustre-MDT0001 complete [ 617.005181] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 619.848959] Key type lgssc unregistered [ 620.177559] LNet: 27039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.193259] LNetError: 27039:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.213968] LNet: Removed LNI 192.168.202.132@tcp [ 621.139956] Key type .llcrypt unregistered [ 621.141605] Key type ._llcrypt unregistered [ 644.568550] Key type ._llcrypt registered [ 644.570847] Key type .llcrypt registered [ 644.719068] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 646.252137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 646.280128] alg: No test for adler32 (adler32-zlib) [ 647.448975] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 647.832258] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 649.560206] Key type lgssc registered [ 651.149755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 659.500175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 659.540794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.188814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.661596] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 670.699657] LustreError: 28435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.744275] LustreError: 28435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 675.721140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 675.820973] LustreError: 28435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 676.062238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 680.164506] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 685.093412] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 688.435506] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 700.740942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 700.905343] LustreError: 30381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.321904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.983632] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 706.442307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 711.105326] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 716.774525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 716.774891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 716.805134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 718.080791] LustreError: 31027:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 718.210518] Lustre: server umount lustre-OST0000 complete [ 721.890685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 721.898572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 721.904756] Lustre: Skipped 1 previous similar message [ 721.907320] Lustre: Skipped 2 previous similar messages [ 727.009533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 727.024207] Lustre: Skipped 2 previous similar messages [ 732.129551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 736.224795] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 736.307348] LustreError: 31228:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 736.315021] LustreError: 31228:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 736.360645] Lustre: server umount lustre-MDT0000 complete [ 737.255398] LustreError: 28432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 739.568403] LustreError: 28415:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338027 with bad export cookie 6995703032297487533 [ 739.583989] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 739.703041] LustreError: 31431:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 739.711141] LustreError: 31431:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 739.916559] Lustre: server umount lustre-MDT0001 complete [ 749.501837] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 752.284259] Key type lgssc unregistered [ 752.519823] LNet: 31962:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.526385] LNetError: 31962:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.536819] LNet: Removed LNI 192.168.202.132@tcp [ 753.277087] Key type .llcrypt unregistered [ 753.279119] Key type ._llcrypt unregistered [ 772.542994] Key type ._llcrypt registered [ 772.544691] Key type .llcrypt registered [ 772.641127] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 773.416660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 773.481096] alg: No test for adler32 (adler32-zlib) [ 774.487139] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 774.710087] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 776.400132] Key type lgssc registered [ 777.372586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 784.258774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 784.293294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.942339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.114850] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 796.130805] LustreError: 33336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 796.163698] LustreError: 33336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 797.497873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 797.790330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 800.855431] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 805.286878] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 808.478196] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 813.848781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 814.220116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.521517] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 825.202352] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 831.630120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 839.140407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 839.143665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.154425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 842.216508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 842.223965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 842.227163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 844.850681] LustreError: 35603:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 845.025604] Lustre: server umount lustre-OST0000 complete [ 848.870301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 848.872854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 848.882610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 848.887287] Lustre: Skipped 1 previous similar message [ 852.449902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.455228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 852.494695] Lustre: Skipped 1 previous similar message [ 854.712404] LustreError: 35805:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 854.742693] LustreError: 35805:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 854.858961] Lustre: server umount lustre-MDT0000 complete [ 857.570173] LustreError: 33336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 857.622641] LustreError: 33336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 858.677552] LustreError: 33316:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338147 with bad export cookie 4635546073785579196 [ 858.695555] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 858.833943] LustreError: 36007:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 858.899041] LustreError: 36007:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 859.178446] Lustre: server umount lustre-MDT0001 complete [ 867.659860] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 870.159065] Key type lgssc unregistered [ 870.419682] LNet: 36539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.430534] LNetError: 36539:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.444701] LNet: Removed LNI 192.168.202.132@tcp [ 871.181201] Key type .llcrypt unregistered [ 871.182826] Key type ._llcrypt unregistered [ 890.156752] Key type ._llcrypt registered [ 890.158367] Key type .llcrypt registered [ 890.271950] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 891.093508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 891.191444] alg: No test for adler32 (adler32-zlib) [ 892.267618] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 892.495905] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 894.160406] Key type lgssc registered [ 895.242894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 902.656407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 902.671064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.037972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.809870] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 914.343876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 914.410225] LustreError: 37914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 914.427384] LustreError: 37914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 914.871963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 918.365162] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 922.335883] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 925.005340] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 930.011128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 930.248880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.719644] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 939.642082] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 948.723151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 955.815131] Lustre: lustre-OST0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 955.875573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.878822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.949846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.953971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.961131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.961143] Lustre: Skipped 1 previous similar message [ 960.654362] LustreError: 40182:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 960.788678] Lustre: server umount lustre-OST0000 complete [ 979.431671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.437703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.440181] Lustre: Skipped 1 previous similar message [ 983.649618] LustreError: 40558:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 983.663394] LustreError: 40558:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 983.746822] Lustre: server umount lustre-MDT0000 complete [ 984.549252] LustreError: 37910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.567910] LustreError: 37910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 986.718459] LustreError: 37894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338275 with bad export cookie 13733060403017631065 [ 986.720419] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 986.725408] LustreError: 37894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 992.930732] LustreError: 40762:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 992.936892] LustreError: 40762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 993.099734] Lustre: server umount lustre-MDT0001 complete [ 1000.840767] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1003.089039] Key type lgssc unregistered [ 1003.372540] LNet: 41296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.383186] LNetError: 41296:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.396471] LNet: Removed LNI 192.168.202.132@tcp [ 1004.131113] Key type .llcrypt unregistered [ 1004.133191] Key type ._llcrypt unregistered [ 1022.727489] Key type ._llcrypt registered [ 1022.728794] Key type .llcrypt registered [ 1022.823815] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1023.642605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1023.684872] alg: No test for adler32 (adler32-zlib) [ 1024.660778] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 1024.867309] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1026.528192] Key type lgssc registered [ 1027.522440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.826355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.838266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.397436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.222372] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1045.587423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1045.872041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1049.212886] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1053.113668] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1055.674738] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1060.967315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1061.282317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.353500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1066.508589] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1071.490450] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1079.205092] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.32@tcp (stopping) [ 1080.191868] LustreError: 44914:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1080.330284] Lustre: server umount lustre-MDT0000 complete [ 1081.826560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.837514] LustreError: 42664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.843294] Lustre: Skipped 1 previous similar message [ 1081.853315] LustreError: 42664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1083.736350] LustreError: 42648:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338372 with bad export cookie 7146124776621935139 [ 1083.743093] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1083.744444] LustreError: 42648:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1084.296867] LustreError: 42664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.309770] LustreError: 42664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1084.333156] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.32@tcp (stopping) [ 1086.946062] LustreError: 42663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1086.953621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.973805] Lustre: Skipped 1 previous similar message [ 1086.977206] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1088.993557] LustreError: 42663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1088.993849] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1094.113291] LustreError: 42663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.113575] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1094.131437] LustreError: 42663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1094.139930] Lustre: Skipped 1 previous similar message [ 1098.208645] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1098.343271] LustreError: 45117:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1098.346843] LustreError: 45117:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1098.523641] Lustre: server umount lustre-MDT0001 complete [ 1112.901857] LustreError: 45341:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1112.904661] LustreError: 45341:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1113.036723] Lustre: server umount lustre-OST0000 complete [ 1125.355248] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1127.630654] Key type lgssc unregistered [ 1127.870450] LNet: 46177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.884655] LNetError: 46177:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.900803] LNet: Removed LNI 192.168.202.132@tcp [ 1128.586487] Key type .llcrypt unregistered [ 1128.588058] Key type ._llcrypt unregistered [ 1147.031079] Key type ._llcrypt registered [ 1147.033604] Key type .llcrypt registered [ 1147.206466] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1147.745127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1147.797434] alg: No test for adler32 (adler32-zlib) [ 1148.796519] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 1149.005662] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1150.672244] Key type lgssc registered [ 1151.395683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1158.374553] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1158.390411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1173.984454] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1179.104223] LustreError: 47520:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 1205.792384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.893730] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1217.747839] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1383.531056] LustreError: 48228:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1383.719207] Lustre: server umount lustre-OST0000 complete [ 1398.305738] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1400.595769] Key type lgssc unregistered [ 1400.917767] LNet: 49066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1400.933601] LNetError: 49066:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1400.960799] LNet: Removed LNI 192.168.202.132@tcp [ 1401.741073] Key type .llcrypt unregistered [ 1401.742964] Key type ._llcrypt unregistered [ 1422.584928] Key type ._llcrypt registered [ 1422.586557] Key type .llcrypt registered [ 1422.705578] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1423.421429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1423.448144] alg: No test for adler32 (adler32-zlib) [ 1424.506368] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 1424.741147] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1426.384168] Key type lgssc registered [ 1427.134381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1433.516310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1433.542626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.976289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.614976] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1445.349643] LustreError: 50452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1445.361840] LustreError: 50452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1446.538712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1446.944529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1451.266925] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1456.423586] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1459.798383] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1466.183222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1466.433082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.086109] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1475.796327] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1480.162984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1480.163288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1484.162284] LustreError: 52694:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1484.310919] Lustre: server umount lustre-OST0000 complete [ 1487.843619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1487.849696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1487.864371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1487.867330] Lustre: Skipped 1 previous similar message [ 1490.401646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.402195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.179308] LustreError: 52896:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1493.183996] LustreError: 52896:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1493.240405] Lustre: server umount lustre-MDT0000 complete [ 1495.523109] LustreError: 50449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1495.540396] LustreError: 50449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1495.771067] LustreError: 50434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338784 with bad export cookie 15963594018013601395 [ 1495.775214] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1495.784450] LustreError: 50434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1495.897237] LustreError: 53098:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1495.900265] LustreError: 53098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1495.998226] Lustre: server umount lustre-MDT0001 complete [ 1504.967201] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1507.351994] Key type lgssc unregistered [ 1507.613303] LNet: 53630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1507.620920] LNetError: 53630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.643181] LNet: Removed LNI 192.168.202.132@tcp [ 1508.284470] Key type .llcrypt unregistered [ 1508.288740] Key type ._llcrypt unregistered [ 1524.924114] Key type ._llcrypt registered [ 1524.925889] Key type .llcrypt registered [ 1524.988388] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1525.674263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1525.696683] alg: No test for adler32 (adler32-zlib) [ 1526.750970] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 1526.954462] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1528.648328] Key type lgssc registered [ 1529.648811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1536.064114] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1536.099133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1551.649360] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1556.770259] LustreError: 54974:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 1583.525345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1588.704531] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1593.146902] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1598.965904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1599.461235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.535980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1603.647983] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1604.578585] LustreError: 55000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1604.600907] LustreError: 55000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1609.707914] LustreError: 55000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1611.784709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1612.205653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1615.583866] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1619.947402] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1623.255325] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1625.057761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1625.070307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.088804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1627.621868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1627.625757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1627.653617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1633.762290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1633.765042] Lustre: Skipped 2 previous similar messages [ 1638.886685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1638.897360] Lustre: Skipped 1 previous similar message [ 1639.392336] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1639.433523] LustreError: 57248:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1639.521094] Lustre: server umount lustre-OST0000 complete [ 1663.457595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.461685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1663.477847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1663.480160] Lustre: Skipped 1 previous similar message [ 1665.859391] LustreError: 57630:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1665.900203] LustreError: 57630:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1666.001066] Lustre: server umount lustre-MDT0000 complete [ 1668.580399] LustreError: 56965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1668.612476] LustreError: 56965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1669.856686] LustreError: 54981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763338958 with bad export cookie 10665055668851251053 [ 1669.861339] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1669.875207] LustreError: 54981:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1676.115048] LustreError: 57832:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1676.134565] LustreError: 57832:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1676.300731] Lustre: server umount lustre-MDT0001 complete [ 1685.235092] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1687.803669] Key type lgssc unregistered [ 1688.083263] LNet: 58365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.095720] LNetError: 58365:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.115592] LNet: Removed LNI 192.168.202.132@tcp [ 1688.932171] Key type .llcrypt unregistered [ 1688.933812] Key type ._llcrypt unregistered [ 1710.729992] Key type ._llcrypt registered [ 1710.731606] Key type .llcrypt registered [ 1710.873423] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1711.836682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.872688] alg: No test for adler32 (adler32-zlib) [ 1712.922227] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 1713.253870] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1714.952174] Key type lgssc registered [ 1716.287032] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1725.153270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1725.185993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.045460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1731.094403] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1737.192842] LustreError: 59760:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.210269] LustreError: 59760:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1738.958449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1739.557221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1743.375517] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1747.521893] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1750.095783] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1756.079445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1756.361572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.922703] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1765.736164] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1771.707871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1782.243438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1782.246877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.257604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1782.267228] Lustre: Skipped 1 previous similar message [ 1782.283709] Lustre: Skipped 1 previous similar message [ 1784.019129] LustreError: 62028:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1784.221830] Lustre: server umount lustre-OST0000 complete [ 1790.950729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1790.962182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.977719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1792.481652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1792.481960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1792.497591] Lustre: Skipped 1 previous similar message [ 1797.607116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.617238] Lustre: Skipped 1 previous similar message [ 1801.702802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.705561] Lustre: Skipped 1 previous similar message [ 1802.208235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1802.273904] LustreError: 62229:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1802.277336] LustreError: 62229:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1802.350278] Lustre: server umount lustre-MDT0000 complete [ 1805.416450] LustreError: 59739:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339093 with bad export cookie 15735091114005710561 [ 1805.430645] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1805.613356] LustreError: 62432:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1805.624434] LustreError: 62432:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1805.781832] Lustre: server umount lustre-MDT0001 complete [ 1814.922374] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1817.699650] Key type lgssc unregistered [ 1818.004493] LNet: 62963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.008524] LNetError: 62963:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.026302] LNet: Removed LNI 192.168.202.132@tcp [ 1818.728405] Key type .llcrypt unregistered [ 1818.729914] Key type ._llcrypt unregistered [ 1845.896646] Key type ._llcrypt registered [ 1845.898191] Key type .llcrypt registered [ 1846.022928] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1846.942818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.982133] alg: No test for adler32 (adler32-zlib) [ 1848.026877] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 1848.209233] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1849.864981] Key type lgssc registered [ 1851.063397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1857.724309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1857.741458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.148393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.081743] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1869.240735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1869.287634] LustreError: 64628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1869.305199] LustreError: 64628:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1869.606768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.401108] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1880.481854] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1884.743937] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1891.951938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1892.250970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1898.022391] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1903.140805] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1904.571618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1909.873394] Lustre: Failing over lustre-MDT0000 [ 1910.023360] LustreError: 66882:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1910.118865] Lustre: server umount lustre-MDT0000 complete [ 1910.756144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1910.779261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1910.786888] LustreError: 66414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.859407] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.862314] LustreError: 66414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.900681] LustreError: 66414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1918.993836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.200473] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.478197] Lustre: *** cfs_fail_loc=135, val=0*** [ 1919.624384] LustreError: 67392:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1919.633534] LustreError: 67392:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.132@tcp: cfg command failed: rc = -2 [ 1919.680208] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1919.680208] [ 1919.692588] LustreError: MGC192.168.202.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1919.725791] LustreError: 67379:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1919.728909] LustreError: 67379:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1919.731853] LustreError: 67379:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1919.738075] LustreError: 64607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339208 with bad export cookie 17568741269005259887 [ 1919.739197] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1919.756784] LustreError: 64607:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1919.772647] Lustre: server umount lustre-MDT0000 complete [ 1919.784824] LustreError: 67379:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1919.975032] LustreError: 64622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.980131] LustreError: 64622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1925.094756] LustreError: 65748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1925.119785] LustreError: 65748:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1926.028510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.499901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.542126] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1931.657754] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1933.404395] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 1935.856360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1935.873413] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1935.915467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1940.456501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1940.460191] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.473131] Lustre: Skipped 1 previous similar message [ 1940.484920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1945.732196] LustreError: 68202:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1945.739454] LustreError: 68202:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1945.902098] Lustre: server umount lustre-OST0000 complete [ 1951.201989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.208595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1951.224615] Lustre: Skipped 1 previous similar message [ 1951.222533] Lustre: Skipped 2 previous similar messages [ 1955.565061] LustreError: 68404:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1955.567513] LustreError: 68404:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1955.649216] Lustre: server umount lustre-MDT0000 complete [ 1956.322243] LustreError: 66414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1956.338186] LustreError: 66414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1959.232745] LustreError: 64609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339247 with bad export cookie 17568741269005260027 [ 1959.241771] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1959.262823] LustreError: 64609:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1959.435560] LustreError: 68605:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1959.449038] LustreError: 68605:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1959.609076] Lustre: server umount lustre-MDT0001 complete [ 1969.152883] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 1971.517850] Key type lgssc unregistered [ 1971.795711] LNet: 69138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.802347] LNetError: 69138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1971.822193] LNet: Removed LNI 192.168.202.132@tcp [ 1972.562389] Key type .llcrypt unregistered [ 1972.563843] Key type ._llcrypt unregistered [ 1992.728291] Key type ._llcrypt registered [ 1992.730096] Key type .llcrypt registered [ 1992.846226] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 1993.506647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.590953] alg: No test for adler32 (adler32-zlib) [ 1994.619821] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 1994.767194] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 1996.424317] Key type lgssc registered [ 1997.448778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2005.153315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2005.195109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2006.666309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.517591] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2016.229958] LustreError: 70513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.252633] LustreError: 70513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2019.654344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2020.201937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2024.066667] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2029.180476] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2033.602528] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2040.816265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2041.356974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.989150] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2051.639497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2052.083498] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2057.320672] Lustre: Failing over lustre-MDT0000 [ 2057.474681] LustreError: 72760:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2057.607867] Lustre: server umount lustre-MDT0000 complete [ 2059.186687] LustreError: 70507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.283658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2061.286689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.315036] LustreError: 70512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2061.794720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.838577] Lustre: Skipped 1 previous similar message [ 2064.275183] LustreError: 70509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.322495] LustreError: 70509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2066.452539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.594866] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.827261] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2066.915094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2066.917542] Lustre: Skipped 2 previous similar messages [ 2067.017590] LustreError: 73269:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 2067.024751] LustreError: 73269:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.132@tcp: cfg command failed: rc = -9 [ 2067.031206] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2067.031206] [ 2067.046541] LustreError: MGC192.168.202.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2067.055943] LustreError: 73257:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2067.070488] LustreError: 73257:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 2067.086056] LustreError: 73257:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 2067.115727] Lustre: server umount lustre-MDT0000 complete [ 2067.123303] LustreError: 73257:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2069.399087] LustreError: 70509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.912791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.051846] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2074.301185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2074.361960] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2074.510221] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2078.340146] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2079.725569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2079.739651] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2079.765810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2086.369254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2086.371694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2086.403403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2089.965569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2089.972426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.079439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2095.082281] Lustre: Skipped 1 previous similar message [ 2100.193645] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2100.209819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2100.244732] Lustre: Skipped 2 previous similar messages [ 2100.315414] LustreError: 74080:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2100.334660] LustreError: 74080:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2100.547886] Lustre: server umount lustre-OST0000 complete [ 2110.434459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.436239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2110.442105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2110.472245] Lustre: Skipped 1 previous similar message [ 2111.616850] LustreError: 74283:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2111.649177] LustreError: 74283:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2111.789636] Lustre: server umount lustre-MDT0000 complete [ 2115.218515] LustreError: 70494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339403 with bad export cookie 14898864171668458013 [ 2115.229894] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2115.249562] LustreError: 70494:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2115.411291] LustreError: 74485:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2115.421995] LustreError: 74485:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2115.690834] Lustre: server umount lustre-MDT0001 complete [ 2124.982549] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2127.555973] Key type lgssc unregistered [ 2127.900619] LNet: 75017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.910427] LNetError: 75017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.935119] LNet: Removed LNI 192.168.202.132@tcp [ 2128.744689] Key type .llcrypt unregistered [ 2128.747916] Key type ._llcrypt unregistered [ 2139.843632] Key type ._llcrypt registered [ 2139.845169] Key type .llcrypt registered [ 2139.938893] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2140.241642] Key type .llcrypt unregistered [ 2140.243410] Key type ._llcrypt unregistered [ 2160.739635] Key type ._llcrypt registered [ 2160.745342] Key type .llcrypt registered [ 2160.856186] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2161.950432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2161.968507] alg: No test for adler32 (adler32-zlib) [ 2163.005360] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 2163.188623] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2164.888176] Key type lgssc registered [ 2166.184680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2173.131225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2173.170900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.765403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2178.102658] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2180.081196] LustreError: 76857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.086656] LustreError: 76857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2185.211282] LustreError: 76858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.015882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2187.300972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2191.705035] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2198.584231] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2202.572147] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2209.733925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2210.101203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.822968] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2218.966975] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2220.351108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2232.804532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2232.807612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2232.841398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2235.877455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2235.877713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2235.879668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.904822] Lustre: Skipped 1 previous similar message [ 2237.625355] LustreError: 79126:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2237.761280] Lustre: server umount lustre-OST0000 complete [ 2243.553345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2243.563371] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.585395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2246.115396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2246.118809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.234370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.240883] Lustre: Skipped 2 previous similar messages [ 2256.352191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2256.470241] LustreError: 79329:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2256.476915] LustreError: 79329:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2256.580290] Lustre: server umount lustre-MDT0000 complete [ 2259.874260] LustreError: 77576:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339548 with bad export cookie 6518041505162541780 [ 2259.884184] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.007811] LustreError: 79531:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2260.015558] LustreError: 79531:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2260.167707] Lustre: server umount lustre-MDT0001 complete [ 2270.497670] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2272.728253] Key type lgssc unregistered [ 2272.995572] LNet: 80063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.003425] LNetError: 80063:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.013832] LNet: Removed LNI 192.168.202.132@tcp [ 2273.703532] Key type .llcrypt unregistered [ 2273.705173] Key type ._llcrypt unregistered [ 2293.827979] Key type ._llcrypt registered [ 2293.830909] Key type .llcrypt registered [ 2293.927883] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2294.653992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.758117] alg: No test for adler32 (adler32-zlib) [ 2295.860901] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 2296.043594] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2297.696232] Key type lgssc registered [ 2298.752079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2305.420369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2305.456571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.992604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2310.162420] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2317.283938] LustreError: 81437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.300075] LustreError: 81437:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2317.356112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2317.670347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2321.505809] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2326.510841] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2329.863814] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2337.286993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2337.842181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2343.362853] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2348.807943] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2352.182167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2362.340135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.341629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2362.348054] Lustre: Skipped 1 previous similar message [ 2362.348991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2362.357205] LustreError: Skipped 1 previous similar message [ 2367.465758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2367.473435] Lustre: Skipped 1 previous similar message [ 2372.578567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2372.581161] Lustre: Skipped 1 previous similar message [ 2373.090066] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2373.158737] LustreError: 83692:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2373.357854] Lustre: server umount lustre-OST0000 complete [ 2377.698080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.742301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.747268] Lustre: Skipped 2 previous similar messages [ 2382.817732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2382.821163] Lustre: Skipped 1 previous similar message [ 2383.241547] LustreError: 83895:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2383.247650] LustreError: 83895:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2383.354089] Lustre: server umount lustre-MDT0000 complete [ 2386.869722] LustreError: 81417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339675 with bad export cookie 8221602798586624125 [ 2386.870371] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2386.882925] LustreError: 81417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2387.029321] LustreError: 84097:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2387.032570] LustreError: 84097:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2387.208903] Lustre: server umount lustre-MDT0001 complete [ 2396.524806] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2398.985984] Key type lgssc unregistered [ 2399.238693] LNet: 84628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.250314] LNetError: 84628:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.270594] LNet: Removed LNI 192.168.202.132@tcp [ 2400.186658] Key type .llcrypt unregistered [ 2400.192476] Key type ._llcrypt unregistered [ 2420.316379] Key type ._llcrypt registered [ 2420.318017] Key type .llcrypt registered [ 2420.416713] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2421.384581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2421.403527] alg: No test for adler32 (adler32-zlib) [ 2422.474332] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 2422.678844] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2424.338118] Key type lgssc registered [ 2425.495821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2433.580908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2433.594324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.012173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.798631] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2445.282555] LustreError: 86002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.374770] LustreError: 86002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2446.943609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2447.239924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2450.221546] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2456.063826] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2460.379657] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2468.459033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2468.733534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2474.781569] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2480.013444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2480.339482] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2492.390090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2492.402785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.427985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2492.908893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2492.915545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.940311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2495.458034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2497.529280] LustreError: 88247:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2497.703586] Lustre: server umount lustre-OST0000 complete [ 2503.652894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2503.666106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.693575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.696620] Lustre: Skipped 1 previous similar message [ 2505.699337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.616142] LustreError: 88449:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2507.618569] LustreError: 88449:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2507.843107] Lustre: server umount lustre-MDT0000 complete [ 2510.819156] LustreError: 86003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2510.837086] LustreError: 86003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2511.684997] LustreError: 88450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763339800 with bad export cookie 10176592231160744153 [ 2511.688301] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2511.695931] LustreError: 88450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2511.883537] LustreError: 88652:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2511.889878] LustreError: 88652:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2512.078343] Lustre: server umount lustre-MDT0001 complete [ 2523.536229] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2527.144820] Key type lgssc unregistered [ 2527.442691] LNet: 89183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2527.448445] LNetError: 89183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2527.461671] LNet: Removed LNI 192.168.202.132@tcp [ 2528.356676] Key type .llcrypt unregistered [ 2528.358340] Key type ._llcrypt unregistered [ 2551.729475] Key type ._llcrypt registered [ 2551.731813] Key type .llcrypt registered [ 2551.863527] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2552.968778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2552.985749] alg: No test for adler32 (adler32-zlib) [ 2554.087290] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 2554.349827] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2556.048169] Key type lgssc registered [ 2557.200957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2565.059197] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2565.081880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2580.640806] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2585.760368] LustreError: 90549:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 2612.448315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.502400] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2624.484244] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2631.077970] LustreError: 91407:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2631.301333] Lustre: server umount lustre-OST0000 complete [ 2643.168796] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 19:38:49 (1763339929) [ 2651.156659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.701742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2656.022992] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2656.741203] LustreError: 92077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2656.773262] LustreError: 92077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2660.839288] LustreError: 92077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.750398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2665.954749] LustreError: 92077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.333625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2670.045557] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2676.236639] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2680.876416] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2688.856361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2689.261604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.368906] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2696.471475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2701.568653] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2712.035527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2712.037114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.042833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2712.060266] Lustre: Skipped 1 previous similar message [ 2717.159104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2717.171616] Lustre: Skipped 1 previous similar message [ 2722.275607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2722.281799] Lustre: Skipped 1 previous similar message [ 2726.368391] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2726.431598] LustreError: 94484:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2726.440718] LustreError: 94484:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2726.570799] Lustre: server umount lustre-OST0000 complete [ 2732.516158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.520961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2732.524298] Lustre: Skipped 1 previous similar message [ 2732.526319] Lustre: Skipped 1 previous similar message [ 2735.867834] LustreError: 94688:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2735.873294] LustreError: 94688:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2735.968564] Lustre: server umount lustre-MDT0000 complete [ 2737.642300] LustreError: 92077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.659551] LustreError: 92077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2739.219534] LustreError: 92058:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340027 with bad export cookie 13443499951551180503 [ 2739.230355] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2739.239317] LustreError: 92058:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2739.348078] LustreError: 94889:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2739.351155] LustreError: 94889:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2739.658281] Lustre: server umount lustre-MDT0001 complete [ 2748.931025] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2751.534158] Key type lgssc unregistered [ 2751.831229] LNet: 95422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2751.835706] LNetError: 95422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2751.856555] LNet: Removed LNI 192.168.202.132@tcp [ 2752.688608] Key type .llcrypt unregistered [ 2752.690119] Key type ._llcrypt unregistered [ 2772.700475] Key type ._llcrypt registered [ 2772.702193] Key type .llcrypt registered [ 2772.783956] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2773.665815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2773.676731] alg: No test for adler32 (adler32-zlib) [ 2774.681879] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 2774.850982] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2776.488206] Key type lgssc registered [ 2777.327421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2782.669458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2782.691834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.089458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2786.934790] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2793.526189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2793.866487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2796.904339] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2801.613962] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2804.364297] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2810.993641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2811.408316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.557588] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2820.905161] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2829.872675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2840.039608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2840.042568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.050511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2840.058137] LustreError: Skipped 1 previous similar message [ 2840.067796] Lustre: Skipped 1 previous similar message [ 2844.616152] LustreError: 99086:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2844.758755] Lustre: server umount lustre-OST0000 complete [ 2850.273475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2850.278485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.287947] Lustre: Skipped 1 previous similar message [ 2850.289705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.289713] Lustre: Skipped 1 previous similar message [ 2854.205653] LustreError: 99288:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2854.217194] LustreError: 99288:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2854.335589] Lustre: server umount lustre-MDT0000 complete [ 2855.397463] LustreError: 96818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2855.404624] LustreError: 96818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2857.834459] LustreError: 96798:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340146 with bad export cookie 3047098631215576128 [ 2857.846224] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2857.854514] LustreError: 96798:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2857.963740] LustreError: 99489:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2857.967695] LustreError: 99489:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2858.210574] Lustre: server umount lustre-MDT0001 complete [ 2868.054872] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 2870.616443] Key type lgssc unregistered [ 2870.861939] LNet: 100021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2870.869553] LNetError: 100021:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2870.885560] LNet: Removed LNI 192.168.202.132@tcp [ 2871.654211] Key type .llcrypt unregistered [ 2871.657180] Key type ._llcrypt unregistered [ 2887.916882] Key type ._llcrypt registered [ 2887.919141] Key type .llcrypt registered [ 2887.995669] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2888.938391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2888.964400] alg: No test for adler32 (adler32-zlib) [ 2889.997925] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 2890.199217] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 2891.840193] Key type lgssc registered [ 2892.870863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2899.167462] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2899.205301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2914.720329] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2919.840355] LustreError: 101243:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 2946.593411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2951.753797] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 2957.255047] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2963.526224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.781801] LustreError: MGC192.168.202.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2963.788318] LustreError: 102155:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2963.796502] LustreError: 102155:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2963.805825] LustreError: 102155:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2963.812712] LustreError: 101251:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340252 with bad export cookie 17654489550685739698 [ 2963.834563] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2963.882759] Lustre: server umount lustre-MDT0000 complete [ 2963.885985] LustreError: 102155:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2970.818618] LustreError: 102587:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2971.021575] Lustre: server umount lustre-OST0000 complete [ 2979.577621] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 2989.476787] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 2997.843312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3003.857603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3009.381367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3015.923694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3022.449886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3022.503790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3022.672227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3022.718739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3022.822766] Lustre: lustre-MDT0000: new disk, initializing [ 3022.927229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3022.941119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3026.400857] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3036.068421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3036.115554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3036.219231] Lustre: 104923:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3036.281298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3036.284290] Lustre: Skipped 1 previous similar message [ 3036.353466] Lustre: lustre-MDT0001: new disk, initializing [ 3036.428209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3036.461645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3036.476141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3040.350440] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3046.960309] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3050.063253] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3056.346455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3056.411881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3056.638716] Lustre: lustre-OST0000: new disk, initializing [ 3056.653097] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3056.728982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3061.635410] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3062.872123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3062.887032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3063.002428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3068.637259] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3072.549272] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3072.835317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3076.304813] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3076.522406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3078.636988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.645872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3078.652092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3083.748754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3083.755882] Lustre: Skipped 2 previous similar messages [ 3084.351164] LustreError: 107864:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3084.362266] LustreError: 107864:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3084.494538] Lustre: server umount lustre-OST0000 complete [ 3088.866068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.877501] Lustre: Skipped 1 previous similar message [ 3088.882331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3093.986366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3093.989480] Lustre: Skipped 2 previous similar messages [ 3094.505664] LustreError: 108066:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3094.522498] LustreError: 108066:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3094.619085] Lustre: server umount lustre-MDT0000 complete [ 3097.830737] LustreError: 104915:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340386 with bad export cookie 17654489550685739789 [ 3097.834410] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3097.847099] LustreError: 104915:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3098.001409] LustreError: 108267:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3098.007265] LustreError: 108267:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3098.233378] Lustre: server umount lustre-MDT0001 complete [ 3105.866326] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 19:46:32 (1763340392) [ 3108.687124] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3121.949430] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 3130.671171] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3140.294854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3146.138706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3151.539900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3157.124533] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3163.371885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3163.444605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.608742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3163.636837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3163.745073] Lustre: lustre-MDT0000: new disk, initializing [ 3163.887302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.903136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3167.594215] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3178.155205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3178.275279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3178.395756] Lustre: 111401:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3178.523650] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3178.528170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3178.542845] Lustre: Skipped 1 previous similar message [ 3178.704486] Lustre: lustre-MDT0001: new disk, initializing [ 3178.894434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3178.960583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3178.976846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3183.559833] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3191.532554] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3195.010893] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3201.547304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3201.649391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3201.924772] Lustre: lustre-OST0000: new disk, initializing [ 3201.927273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3202.060556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3207.284233] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3208.771819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3208.780775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3208.891718] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3214.142551] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3218.238954] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3218.482758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3222.849149] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3223.025497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3229.667790] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3229.675640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.678183] LustreError: Skipped 2 previous similar messages [ 3229.695071] Lustre: Skipped 1 previous similar message [ 3229.700081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3239.392271] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3239.431149] LustreError: 114343:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3239.467973] LustreError: 114343:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3239.704365] Lustre: server umount lustre-OST0000 complete [ 3245.032142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.040541] Lustre: Skipped 2 previous similar messages [ 3249.770673] LustreError: 114545:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3249.787553] LustreError: 114545:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3249.945175] Lustre: server umount lustre-MDT0000 complete [ 3253.591228] LustreError: 111394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340541 with bad export cookie 17654489550685740671 [ 3253.600918] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3253.616275] LustreError: 111394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3253.944177] Lustre: server umount lustre-MDT0001 complete [ 3261.046610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.361410] LustreError: 115226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.373842] LustreError: 115226:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3261.416307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.662370] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3266.530105] LustreError: 115226:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.653772] LustreError: 115225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3272.519601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3277.441777] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3284.072129] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3288.263609] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3295.408318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3302.039778] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3306.875738] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3313.829710] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3318.753402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3318.759896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.768928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3318.772640] Lustre: Skipped 5 previous similar messages [ 3321.831434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3323.338596] LustreError: 117568:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3323.347793] LustreError: 117568:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3323.450260] Lustre: server umount lustre-OST0000 complete [ 3326.948216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3326.968942] Lustre: Skipped 2 previous similar messages [ 3333.064300] Lustre: server umount lustre-MDT0000 complete [ 3336.655284] LustreError: 115206:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340624 with bad export cookie 17654489550685741553 [ 3336.662531] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3336.665682] LustreError: 115206:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3337.005634] Lustre: server umount lustre-MDT0001 complete [ 3345.900256] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 3348.416123] Key type lgssc unregistered [ 3348.700634] LNet: 118503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3348.721868] LNetError: 118503:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3348.752288] LNet: Removed LNI 192.168.202.132@tcp [ 3349.595804] Key type .llcrypt unregistered [ 3349.597809] Key type ._llcrypt unregistered [ 3364.811771] Key type ._llcrypt registered [ 3364.812946] Key type .llcrypt registered [ 3364.904472] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 3377.425733] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 3377.991159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3378.021226] alg: No test for adler32 (adler32-zlib) [ 3379.068110] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 3379.226600] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 3380.872223] Key type lgssc registered [ 3381.821922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3389.261933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3395.456487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3400.324654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3405.732054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3412.161716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3412.208672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3412.222894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.425944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3413.473807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3413.547680] Lustre: lustre-MDT0000: new disk, initializing [ 3413.674516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.693785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3417.288031] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3426.120954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3426.202934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3426.339944] Lustre: 121627:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3426.435859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3426.440797] Lustre: Skipped 1 previous similar message [ 3426.509465] Lustre: lustre-MDT0001: new disk, initializing [ 3426.629390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3426.677403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3426.689520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3430.527619] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3436.555949] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3439.491782] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3444.874384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3444.969159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3445.204795] Lustre: lustre-OST0000: new disk, initializing [ 3445.211508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3445.279301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3449.528940] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3454.515255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3454.525858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3454.629960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3455.707991] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3459.558565] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3459.786422] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3464.212496] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3464.411779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3470.318952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3470.320506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3470.327731] LustreError: Skipped 1 previous similar message [ 3470.333044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3470.353513] Lustre: Skipped 1 previous similar message [ 3472.264107] LustreError: 124563:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3472.383693] Lustre: server umount lustre-OST0000 complete [ 3477.985304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3477.994823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3478.015914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3478.025585] Lustre: Skipped 1 previous similar message [ 3480.550267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3480.554188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.675216] LustreError: 124765:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3482.678592] LustreError: 124765:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3482.779551] Lustre: server umount lustre-MDT0000 complete [ 3486.126219] LustreError: 121619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340774 with bad export cookie 4481912628140923611 [ 3486.130256] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3486.141438] LustreError: 121619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3486.265375] LustreError: 124967:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3486.268792] LustreError: 124967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3486.489321] Lustre: server umount lustre-MDT0001 complete [ 3493.764701] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 19:53:00 (1763340780) [ 3499.745676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.257657] LustreError: 125641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3500.272446] LustreError: 125641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3500.340335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3504.961959] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3505.646405] LustreError: 125641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.744609] LustreError: 125641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3513.714884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3514.071720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3517.576520] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3522.377087] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3525.864545] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3529.698756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3529.703887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.711051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3529.715598] Lustre: Skipped 1 previous similar message [ 3534.013851] LustreError: 126983:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3534.018639] LustreError: 126983:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3534.148306] Lustre: server umount lustre-MDT0000 complete [ 3535.843059] LustreError: 125640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3535.856952] LustreError: 125640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3537.004884] LustreError: 125620:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340825 with bad export cookie 4481912628140924493 [ 3537.010331] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3537.024742] LustreError: 125620:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3537.317578] Lustre: server umount lustre-MDT0001 complete [ 3543.627183] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 19:53:50 (1763340830) [ 3547.989748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3563.552834] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3568.672547] LustreError: 127829:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 3594.465216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3599.702166] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3604.657852] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3606.600073] LustreError: 128494:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3606.602563] LustreError: 128494:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3606.752414] Lustre: server umount lustre-OST0000 complete [ 3614.421819] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 19:55:01 (1763340901) [ 3620.237077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.576283] LustreError: 129166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3620.645057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.671641] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3630.055991] LustreError: 129165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3630.073775] LustreError: 129165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3631.565903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3636.149971] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3642.312350] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3646.170828] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3651.766983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3657.430852] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3662.841072] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3673.060522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3673.064054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.068797] Lustre: Skipped 1 previous similar message [ 3673.071782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3673.074477] Lustre: Skipped 2 previous similar messages [ 3679.221659] LustreError: 131412:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3679.229321] LustreError: 131412:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3679.319912] Lustre: server umount lustre-MDT0000 complete [ 3680.750869] LustreError: 129166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.771493] LustreError: 129166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3682.855817] LustreError: 129884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340971 with bad export cookie 4481912628140925158 [ 3682.863546] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3682.922914] LustreError: 129884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3683.152628] Lustre: server umount lustre-MDT0001 complete [ 3697.093369] LustreError: 131815:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3697.097557] LustreError: 131815:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3697.244709] Lustre: server umount lustre-OST0000 complete [ 3704.270765] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 19:56:31 (1763340991) [ 3710.199207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.839645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3710.842347] Lustre: Skipped 2 previous similar messages [ 3714.140836] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3716.066068] LustreError: 132487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.082231] LustreError: 132487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3721.195313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3724.966744] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3729.116853] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3731.547975] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3736.609427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3741.582116] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3746.456536] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3750.103759] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3756.343120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 3756.595424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 3759.510817] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3759.711130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3761.634592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3761.636515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.645241] LustreError: Skipped 1 previous similar message [ 3761.648339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3761.650179] Lustre: Skipped 3 previous similar messages [ 3761.658398] Lustre: Skipped 3 previous similar messages [ 3763.223373] LustreError: 135271:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3763.226604] LustreError: 135271:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3763.307324] Lustre: server umount lustre-OST0000 complete [ 3772.686136] Lustre: server umount lustre-MDT0000 complete [ 3776.123843] LustreError: 135273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341064 with bad export cookie 4481912628140926222 [ 3776.127493] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3776.135973] LustreError: 135273:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3776.428363] Lustre: server umount lustre-MDT0001 complete [ 3783.829512] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 19:57:50 (1763341070) [ 3789.125149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3804.643732] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3809.760630] LustreError: 136317:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 3835.488419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3835.491317] Lustre: Skipped 2 previous similar messages [ 3839.960963] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3845.180733] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3850.750139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3851.130639] LustreError: 136344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.160687] LustreError: 136344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3851.298294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3854.785253] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3862.116168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3865.601824] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3869.976309] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3872.867527] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3875.436121] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3875.593745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3878.398094] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3878.546234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3879.906335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3879.910142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3879.914166] Lustre: Skipped 4 previous similar messages [ 3879.916399] Lustre: Skipped 6 previous similar messages [ 3894.240083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3894.301129] LustreError: 139096:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3894.303562] LustreError: 139096:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3894.371993] Lustre: server umount lustre-MDT0000 complete [ 3897.109241] LustreError: 136324:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341185 with bad export cookie 4481912628140927251 [ 3897.126167] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3897.419160] Lustre: server umount lustre-MDT0001 complete [ 3910.777721] Lustre: server umount lustre-OST0000 complete [ 3918.123602] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 20:00:04 (1763341204) [ 3923.416177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3939.040667] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3944.160452] LustreError: 140145:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 3969.888993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.895220] Lustre: Skipped 2 previous similar messages [ 3974.585277] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3979.149909] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3985.348124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3985.703624] LustreError: 140172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3985.728068] LustreError: 140172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3985.833058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3988.867062] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 3995.988724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3999.538770] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4004.245873] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4007.194061] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4013.351719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4013.431943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4013.604996] Lustre: lustre-OST0001: new disk, initializing [ 4013.608800] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4018.904736] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4019.237227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4019.253833] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4019.364818] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4025.522125] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4029.155177] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4029.332641] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4032.534520] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4032.750420] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4034.538677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4034.545050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4034.547473] Lustre: Skipped 1 previous similar message [ 4034.552789] Lustre: Skipped 6 previous similar messages [ 4040.574819] LustreError: 143885:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4040.583267] LustreError: 143885:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4040.635141] Lustre: server umount lustre-OST0000 complete [ 4044.778514] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4049.696824] Lustre: server umount lustre-OST0001 complete [ 4059.411970] Lustre: server umount lustre-MDT0000 complete [ 4063.068534] LustreError: 140153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341351 with bad export cookie 4481912628140928196 [ 4063.070281] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4063.078931] LustreError: 140153:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4063.329745] Lustre: server umount lustre-MDT0001 complete [ 4068.111381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4071.016836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4073.898152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4076.791583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4081.279755] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 20:02:48 (1763341368) [ 4082.436121] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4083.635359] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4085.014766] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 20:02:51 (1763341371) [ 4090.779090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.060348] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4091.093326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4094.699987] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4101.761515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4101.817096] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4101.847759] Lustre: 146129:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4105.316058] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4110.128372] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4113.699642] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4119.884333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4120.051491] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4125.478400] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4125.800300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4129.942076] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4132.917563] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4133.116935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4136.355720] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4136.552465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4141.035173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4141.042091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.054710] Lustre: Skipped 4 previous similar messages [ 4141.062754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4141.064556] Lustre: Skipped 7 previous similar messages [ 4152.288972] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4152.530240] Lustre: server umount lustre-OST0000 complete [ 4171.817377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4177.587106] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4182.233697] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4182.385698] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4182.405669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4182.408931] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4182.434699] Lustre: Skipped 1 previous similar message [ 4185.831053] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4186.029177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4189.102420] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4189.252648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4192.065825] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d30448e6000.ost_server_uuid 50 [ 4193.193319] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d30448e6000.ost_server_uuid in FULL state after 0 sec [ 4193.325328] LustreError: 146136:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.202.32@tcp) returned error from blocking AST (req@000000004cfeb241 x1848996703470848 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000458b2cc9/0x3e32f3ca02ad7617 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.32@tcp remote: 0x86df6292ad8124bb expref: 5 pid: 146136 timeout: 4293 lvb_type: 0 [ 4193.343606] LustreError: lustre-MDT0000: A client on nid 192.168.202.32@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4193.348612] LustreError: 146125:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.32@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000458b2cc9/0x3e32f3ca02ad7617 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.32@tcp remote: 0x86df6292ad8124bb expref: 6 pid: 146136 timeout: 0 lvb_type: 0 [ 4197.857136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4197.860961] LustreError: Skipped 2 previous similar messages [ 4202.212568] Lustre: server umount lustre-OST0000 complete [ 4210.885297] Lustre: server umount lustre-MDT0000 complete [ 4213.224050] LustreError: 146120:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341501 with bad export cookie 4481912628140929372 [ 4213.227680] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4213.229978] LustreError: 146120:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4219.582964] Lustre: server umount lustre-MDT0001 complete [ 4226.840578] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4228.676604] Key type lgssc unregistered [ 4228.878690] LNet: 151617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4228.891276] LNetError: 151617:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4228.909639] LNet: Removed LNI 192.168.202.132@tcp [ 4229.567466] Key type .llcrypt unregistered [ 4229.569218] Key type ._llcrypt unregistered [ 4245.748619] Key type ._llcrypt registered [ 4245.751880] Key type .llcrypt registered [ 4245.845633] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4246.407161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4246.482345] alg: No test for adler32 (adler32-zlib) [ 4247.568437] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 4247.753295] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4249.424249] Key type lgssc registered [ 4250.231840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4256.323226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4256.348348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4257.698607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4260.316608] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4266.512610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4266.795240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4269.461408] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4272.716206] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4274.925984] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4279.241100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4279.527536] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4283.651806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4283.765360] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4287.714977] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4296.397313] Lustre: Failing over lustre-MDT0000 [ 4296.522288] LustreError: 155236:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4296.613537] Lustre: server umount lustre-MDT0000 complete [ 4297.189242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4297.189811] LustreError: 153416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4297.212576] Lustre: Skipped 2 previous similar messages [ 4297.230069] LustreError: 153416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4302.314076] LustreError: 152987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4302.326152] LustreError: 152987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4307.429695] LustreError: 152991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.449393] LustreError: 152991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4308.375216] LustreError: 152971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.32@tcp arrived at 1763341596 with bad export cookie 7587949448597715595 [ 4308.381716] LustreError: 152971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4312.546258] LustreError: 152985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4312.574239] LustreError: 152985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4313.570952] Lustre: 152300:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763341585/real 1763341585] req@000000007ab9d390 x1848997614474752/t0(0) o400->MGC192.168.202.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763341601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4313.597810] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4314.847666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4318.691845] LustreError: 152985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4318.708048] LustreError: 152985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4322.789054] LustreError: 152298:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000087ac605c x1848997614480000/t0(0) o250->MGC192.168.202.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4323.007713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.047463] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4326.543197] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4328.422384] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4328.431216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4328.463477] Lustre: lustre-MDT0000: Denying connection for new client 90d4357a-981f-4916-a58a-3073a79b05e6 (at 192.168.202.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 4338.145664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4338.149428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.158063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.669538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4342.684218] LustreError: 156171:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4342.724460] LustreError: 156171:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4342.907741] Lustre: server umount lustre-OST0000 complete [ 4346.298445] LustreError: 156372:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4346.308710] Lustre: 155736:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4346.314522] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4346.331281] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4346.344436] LustreError: 155736:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006ef9cf4c x1848997614493952/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4346.356187] LustreError: 155736:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4346.361974] LustreError: 155736:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4346.369754] Lustre: lustre-MDT0000: Recovery over after 0:18, of 2 clients 0 recovered and 2 were evicted. [ 4346.528853] LustreError: 156372:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4346.554677] LustreError: 156372:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4346.683872] Lustre: server umount lustre-MDT0000 complete [ 4348.900617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.909428] Lustre: Skipped 1 previous similar message [ 4348.921209] LustreError: 152985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4348.953335] LustreError: 152985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4350.019855] LustreError: 152298:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006e82cd6c x1848997614494720/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4350.020918] LustreError: 155237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341638 with bad export cookie 7587949448597715728 [ 4350.021758] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4350.036996] LustreError: 155237:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4352.161554] LustreError: 156574:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4352.165547] LustreError: 156574:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4352.287675] Lustre: server umount lustre-MDT0001 complete [ 4362.028513] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4364.683923] Key type lgssc unregistered [ 4364.970534] LNet: 157106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4364.982945] LNetError: 157106:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4365.004892] LNet: Removed LNI 192.168.202.132@tcp [ 4365.682428] Key type .llcrypt unregistered [ 4365.684079] Key type ._llcrypt unregistered [ 4384.814650] Key type ._llcrypt registered [ 4384.818686] Key type .llcrypt registered [ 4384.879768] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4385.641766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4385.683244] alg: No test for adler32 (adler32-zlib) [ 4386.743580] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 4386.913058] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4388.600201] Key type lgssc registered [ 4389.743984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4397.195558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4397.221988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.066730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4404.391994] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4409.324370] LustreError: 158479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.330850] LustreError: 158479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4412.486734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4412.997561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4416.969719] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4422.389709] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4426.387795] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4432.758276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4433.152766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.479860] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4442.106164] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4444.414464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4453.862445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4453.873682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.886197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4454.882766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4454.889281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4454.894837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.912946] Lustre: Skipped 1 previous similar message [ 4457.788139] LustreError: 160724:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4457.927692] Lustre: server umount lustre-OST0000 complete [ 4464.097165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4464.107675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4464.121759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4470.241853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4470.247281] Lustre: Skipped 3 previous similar messages [ 4475.363454] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4475.364782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4475.381242] Lustre: Skipped 1 previous similar message [ 4475.445158] LustreError: 160927:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4475.449035] LustreError: 160927:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4475.535692] Lustre: server umount lustre-MDT0000 complete [ 4478.481818] LustreError: 158458:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341766 with bad export cookie 11490200985114159723 [ 4478.495368] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.598372] LustreError: 161132:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4478.602488] LustreError: 161132:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4478.760836] Lustre: server umount lustre-MDT0001 complete [ 4487.562374] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4489.736566] Key type lgssc unregistered [ 4490.002631] LNet: 161662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4490.012570] LNetError: 161662:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4490.032330] LNet: Removed LNI 192.168.202.132@tcp [ 4490.637303] Key type .llcrypt unregistered [ 4490.639401] Key type ._llcrypt unregistered [ 4519.281680] Key type ._llcrypt registered [ 4519.285784] Key type .llcrypt registered [ 4519.435293] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4520.863768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4520.928795] alg: No test for adler32 (adler32-zlib) [ 4521.981721] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 4522.197589] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4523.896279] Key type lgssc registered [ 4524.840724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4531.080488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4531.105644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4532.680656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4535.973795] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4541.923521] LustreError: 163345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.945860] LustreError: 163345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4542.854363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4543.146420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4546.570564] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4551.465586] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4555.215952] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4560.811121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4561.053958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4565.967948] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4570.998166] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4577.457991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 4582.891447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.897952] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4582.917405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4583.234177] LustreError: 165592:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4583.333875] Lustre: server umount lustre-OST0000 complete [ 4588.005909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.016209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4588.031997] Lustre: Skipped 1 previous similar message [ 4588.042451] Lustre: Skipped 2 previous similar messages [ 4592.546978] LustreError: 165793:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4592.550396] LustreError: 165793:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4592.621127] Lustre: server umount lustre-MDT0000 complete [ 4593.121760] LustreError: 163345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4593.132026] LustreError: 163345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4595.727459] LustreError: 165794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341884 with bad export cookie 6270797447038283260 [ 4595.736751] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4595.741974] LustreError: 165794:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4595.850492] LustreError: 165995:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4595.867303] LustreError: 165995:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4596.027350] Lustre: server umount lustre-MDT0001 complete [ 4604.448204] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4607.053464] Key type lgssc unregistered [ 4607.338137] LNet: 166527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4607.349028] LNetError: 166527:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4607.363107] LNet: Removed LNI 192.168.202.132@tcp [ 4608.088483] Key type .llcrypt unregistered [ 4608.091139] Key type ._llcrypt unregistered [ 4626.191037] Key type ._llcrypt registered [ 4626.192533] Key type .llcrypt registered [ 4626.303362] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4626.932119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4626.949186] alg: No test for adler32 (adler32-zlib) [ 4628.118074] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 4628.393632] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4630.096228] Key type lgssc registered [ 4631.294154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4642.492116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4642.504887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.784895] Lustre: *** cfs_fail_loc=135, val=0*** [ 4643.853069] LustreError: 168012:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 4643.866527] LustreError: 168012:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.202.132@tcp: cfg command failed: rc = -2 [ 4643.871986] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 4643.871986] [ 4643.882072] LustreError: MGC192.168.202.132@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4643.902732] LustreError: 167971:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4643.915336] LustreError: 167971:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4643.925305] LustreError: 167971:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 4643.944551] LustreError: 167971:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4644.011875] Lustre: server umount lustre-MDT0000 complete [ 4644.015400] LustreError: 167971:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4650.811257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.230645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4654.389407] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4660.774811] LustreError: 168385:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.790225] LustreError: 168385:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4661.446076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4661.820783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4665.787592] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4671.190492] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4675.083187] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4681.195262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4681.216366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4681.233764] Lustre: Skipped 1 previous similar message [ 4683.758343] LustreError: 169726:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4683.793770] LustreError: 169726:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4684.025880] Lustre: server umount lustre-MDT0000 complete [ 4686.307864] LustreError: 168380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4686.319830] LustreError: 168380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4687.334468] LustreError: 169096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.360238] LustreError: 169096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4687.678664] LustreError: 168364:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341976 with bad export cookie 18201991213212301594 [ 4687.686605] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4687.693804] LustreError: 168364:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4687.847870] LustreError: 169928:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4687.850823] LustreError: 169928:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4688.072921] Lustre: server umount lustre-MDT0001 complete [ 4699.048732] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4702.699260] Key type lgssc unregistered [ 4703.234263] LNet: 170461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4703.256165] LNetError: 170461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4704.298281] LNet: Removed LNI 192.168.202.132@tcp [ 4705.105793] Key type .llcrypt unregistered [ 4705.107843] Key type ._llcrypt unregistered [ 4722.204452] Key type ._llcrypt registered [ 4722.206063] Key type .llcrypt registered [ 4722.287481] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4722.466633] Key type .llcrypt unregistered [ 4722.468500] Key type ._llcrypt unregistered [ 4735.375568] Key type ._llcrypt registered [ 4735.377104] Key type .llcrypt registered [ 4735.449484] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4736.202648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4736.221149] alg: No test for adler32 (adler32-zlib) [ 4737.270855] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 4737.530548] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4739.208182] Key type lgssc registered [ 4740.497829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4748.151191] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4748.168481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4763.680369] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4768.800190] LustreError: 172532:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.132@tcp: failed processing log, type 4: rc = -110 [ 4795.494459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4800.536213] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4805.001991] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4810.796464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4811.167767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.224847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 4814.404786] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4816.361344] LustreError: 172559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4816.388272] LustreError: 172559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4821.480362] LustreError: 172559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4821.717545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4822.082723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4826.790324] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4832.278876] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4835.696388] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4838.472130] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4850.660436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4850.669549] Lustre: Skipped 1 previous similar message [ 4850.673465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4854.454376] LustreError: 175208:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 4854.550110] Lustre: server umount lustre-OST0000 complete [ 4860.903497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.920260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4860.922453] Lustre: Skipped 1 previous similar message [ 4864.453579] LustreError: 175411:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 4864.470768] LustreError: 175411:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4864.524710] Lustre: server umount lustre-MDT0000 complete [ 4866.019751] LustreError: 173463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4866.042087] LustreError: 173463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4868.144066] LustreError: 174859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342156 with bad export cookie 6086059444334706371 [ 4868.149111] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4868.154451] LustreError: 174859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4868.324209] LustreError: 175612:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4868.326724] LustreError: 175612:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4868.608473] Lustre: server umount lustre-MDT0001 complete [ 4879.993765] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 4882.740621] Key type lgssc unregistered [ 4883.009694] LNet: 176143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4883.019319] LNetError: 176143:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4883.034109] LNet: Removed LNI 192.168.202.132@tcp [ 4883.732780] Key type .llcrypt unregistered [ 4883.737597] Key type ._llcrypt unregistered [ 4903.709476] Key type ._llcrypt registered [ 4903.710785] Key type .llcrypt registered [ 4903.800714] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 4904.700522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4904.755717] alg: No test for adler32 (adler32-zlib) [ 4905.777673] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 4905.977221] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 4907.648175] Key type lgssc registered [ 4908.728904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4915.528815] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4915.550761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4917.059258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4920.495350] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4927.458799] LustreError: 177516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4927.483452] LustreError: 177516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4928.096783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4928.486194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4931.642587] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4935.766773] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4938.849591] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4944.944472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4945.214435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.132615] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4955.647632] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4962.619193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 4969.195079] Lustre: Failing over lustre-MDT0000 [ 4969.349374] LustreError: 179835:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4969.441911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4969.449520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.487263] LustreError: 177516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.498264] Lustre: server umount lustre-MDT0000 complete [ 4971.431145] LustreError: 177511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4973.027800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.033372] Lustre: Skipped 1 previous similar message [ 4976.523795] LustreError: 177512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.550835] LustreError: 177512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4981.643395] LustreError: 177512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4981.670197] LustreError: 177512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4987.772070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.851961] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4988.060982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.105239] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4991.589280] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 4991.884124] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4993.515228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4993.538863] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4993.566721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 4995.362835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 4996.852053] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5003.749814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5003.754266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5003.767441] Lustre: Skipped 1 previous similar message [ 5003.779281] Lustre: Skipped 1 previous similar message [ 5005.712644] LustreError: 180970:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5005.724618] LustreError: 180970:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5005.948840] Lustre: server umount lustre-OST0000 complete [ 5013.986374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.987726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5014.013322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5015.857349] LustreError: 181175:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5015.860666] LustreError: 181175:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5015.988677] Lustre: server umount lustre-MDT0000 complete [ 5019.115079] LustreError: 178229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5019.125419] LustreError: 178229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5020.267220] LustreError: 177497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342308 with bad export cookie 14156759369172564576 [ 5020.269586] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5020.288643] LustreError: 177497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5020.503486] LustreError: 181377:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5020.506468] LustreError: 181377:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5020.694953] Lustre: server umount lustre-MDT0001 complete [ 5030.720257] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5033.330195] Key type lgssc unregistered [ 5033.648523] LNet: 181909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5033.656291] LNetError: 181909:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5033.671151] LNet: Removed LNI 192.168.202.132@tcp [ 5034.486710] Key type .llcrypt unregistered [ 5034.488329] Key type ._llcrypt unregistered [ 5058.629163] Key type ._llcrypt registered [ 5058.630907] Key type .llcrypt registered [ 5058.747097] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5059.367996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5059.400455] alg: No test for adler32 (adler32-zlib) [ 5060.413322] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 5060.613922] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5062.280197] Key type lgssc registered [ 5063.233210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5069.806954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5069.827137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.191847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5074.260058] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5081.435244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5081.570365] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5081.758901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5085.252585] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5089.846229] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5092.557812] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5097.535928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5097.842611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5102.729452] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5107.945844] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5116.278613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5121.128260] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5128.937218] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5136.927070] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5147.109813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5147.111225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5147.161606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5152.233310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5152.235714] Lustre: Skipped 1 previous similar message [ 5152.284721] LustreError: 185729:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5152.430353] Lustre: server umount lustre-OST0000 complete [ 5157.347704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.358058] Lustre: Skipped 1 previous similar message [ 5157.362836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5157.368064] Lustre: Skipped 1 previous similar message [ 5162.468103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5162.480066] Lustre: Skipped 1 previous similar message [ 5162.489692] LustreError: 185936:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5162.505893] LustreError: 185936:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5162.672815] Lustre: server umount lustre-MDT0000 complete [ 5166.464066] LustreError: 183278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342454 with bad export cookie 8206175155023389380 [ 5166.465976] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5166.486022] LustreError: 183278:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5166.621826] LustreError: 186139:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5166.628588] LustreError: 186139:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5166.818938] Lustre: server umount lustre-MDT0001 complete [ 5178.655415] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5181.532702] Key type lgssc unregistered [ 5181.799320] LNet: 186670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5181.806208] LNetError: 186670:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5181.823190] LNet: Removed LNI 192.168.202.132@tcp [ 5182.516100] Key type .llcrypt unregistered [ 5182.517593] Key type ._llcrypt unregistered [ 5209.223603] Key type ._llcrypt registered [ 5209.225444] Key type .llcrypt registered [ 5209.411885] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5210.694510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5210.702711] alg: No test for adler32 (adler32-zlib) [ 5212.103976] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 5212.475469] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5214.203925] Key type lgssc registered [ 5215.486833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5223.612939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5223.648065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.106799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5229.150626] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5235.684181] LustreError: 188064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5235.717314] LustreError: 188064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5237.798601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5238.127837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5241.618202] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5246.701299] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5250.383757] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5256.772222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5257.101902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5262.900874] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5268.145766] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5271.480977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 5277.741268] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 5286.534552] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5294.585538] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 5303.254550] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 5312.489859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5312.496649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5312.501053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5317.383804] LustreError: 191705:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5317.527375] Lustre: server umount lustre-OST0000 complete [ 5322.726261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.731695] Lustre: Skipped 1 previous similar message [ 5322.734290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5322.769039] Lustre: Skipped 2 previous similar messages [ 5327.113484] LustreError: 191906:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5327.132406] LustreError: 191906:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5327.446925] Lustre: server umount lustre-MDT0000 complete [ 5327.843607] LustreError: 188779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.861758] LustreError: 188779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5331.801193] LustreError: 190411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342620 with bad export cookie 14417108404336591509 [ 5331.804940] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5331.815383] LustreError: 190411:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5332.066273] LustreError: 192108:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5332.078174] LustreError: 192108:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5332.298643] Lustre: server umount lustre-MDT0001 complete [ 5341.190095] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5344.057691] Key type lgssc unregistered [ 5344.397866] LNet: 192638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5344.412556] LNetError: 192638:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5345.449077] LNet: Removed LNI 192.168.202.132@tcp [ 5346.421184] Key type .llcrypt unregistered [ 5346.422591] Key type ._llcrypt unregistered [ 5368.702461] Key type ._llcrypt registered [ 5368.704090] Key type .llcrypt registered [ 5368.798871] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5369.582789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5369.608320] alg: No test for adler32 (adler32-zlib) [ 5370.672835] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 5370.898034] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5372.560192] Key type lgssc registered [ 5373.568592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5381.160516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5381.188805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5382.597158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5385.815151] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5391.847823] LustreError: 194031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5391.886324] LustreError: 194031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5394.308116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5394.686711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5398.345554] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5404.443654] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5408.112681] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5414.487666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5414.979418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5421.660371] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5427.273392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 5427.844404] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5442.545366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.547672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5442.579210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5442.624300] Lustre: Skipped 1 previous similar message [ 5446.892210] LustreError: 196430:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5447.075677] Lustre: server umount lustre-OST0000 complete [ 5451.235536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5451.249909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.269045] Lustre: Skipped 1 previous similar message [ 5451.286889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5452.775151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.776046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5452.797639] Lustre: Skipped 1 previous similar message [ 5457.329356] LustreError: 196632:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5457.332362] LustreError: 196632:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5457.435108] Lustre: server umount lustre-MDT0000 complete [ 5457.897746] LustreError: 194745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5457.926442] LustreError: 194745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5461.042559] LustreError: 196197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342749 with bad export cookie 7847097628542431243 [ 5461.044677] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5461.052695] LustreError: 196197:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5461.247400] LustreError: 196834:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5461.282389] LustreError: 196834:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5461.452474] Lustre: server umount lustre-MDT0001 complete [ 5470.643782] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5472.984548] Key type lgssc unregistered [ 5473.252401] LNet: 197365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5473.256513] LNetError: 197365:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5473.282047] LNet: Removed LNI 192.168.202.132@tcp [ 5474.115728] Key type .llcrypt unregistered [ 5474.133451] Key type ._llcrypt unregistered [ 5491.253758] Key type ._llcrypt registered [ 5491.255237] Key type .llcrypt registered [ 5491.343256] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5492.489560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5492.525953] alg: No test for adler32 (adler32-zlib) [ 5493.518995] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 5493.731544] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5495.408518] Key type lgssc registered [ 5496.488276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5504.609946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5504.627526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.210051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.774139] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5515.750487] LustreError: 198608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5515.764167] LustreError: 198608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5518.449384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5519.001629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5523.158315] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5528.870203] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5533.065030] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5541.077874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5541.660491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.174439] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5551.946420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 5552.337711] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5567.459420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5567.462133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5567.478494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5567.485417] Lustre: Skipped 1 previous similar message [ 5570.969987] LustreError: 201003:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5571.161723] Lustre: server umount lustre-OST0000 complete [ 5575.141023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5575.144313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5575.168932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5575.178491] Lustre: Skipped 1 previous similar message [ 5577.698909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5577.701248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.040607] LustreError: 201205:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5581.049817] LustreError: 201205:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5581.184285] Lustre: server umount lustre-MDT0000 complete [ 5582.828537] LustreError: 198609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5582.859980] LustreError: 198609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5584.330578] LustreError: 201206:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342872 with bad export cookie 8806336284654024302 [ 5584.334051] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5584.362213] LustreError: 201206:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5584.468087] LustreError: 201408:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5584.476074] LustreError: 201408:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5584.639789] Lustre: server umount lustre-MDT0001 complete [ 5594.515320] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5597.537613] Key type lgssc unregistered [ 5597.897944] LNet: 201940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5597.913206] LNetError: 201940:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5597.939813] LNet: Removed LNI 192.168.202.132@tcp [ 5599.057543] Key type .llcrypt unregistered [ 5599.061598] Key type ._llcrypt unregistered [ 5617.165277] Key type ._llcrypt registered [ 5617.166953] Key type .llcrypt registered [ 5617.290356] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5618.507882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5618.603608] alg: No test for adler32 (adler32-zlib) [ 5619.728761] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 5619.975080] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5621.672389] Key type lgssc registered [ 5622.914947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5630.692638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5630.715292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.278744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.466963] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5641.706129] LustreError: 203186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.732460] LustreError: 203186:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5645.500472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5646.196955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5651.458739] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5657.143436] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5660.970404] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5667.664677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5668.122603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5673.495288] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5677.364800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 5678.652548] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5687.679655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5687.726485] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5687.891921] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5693.339036] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5699.178832] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5700.831678] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5712.596916] Lustre: Permanently deactivating lustre-OST0001 [ 5712.641648] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5722.203660] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5732.368390] Lustre: Permanently reactivating lustre-OST0001 [ 5732.390419] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5732.393859] Lustre: Skipped 2 previous similar messages [ 5738.061518] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5738.099585] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5738.118468] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5738.136603] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5738.181785] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5748.706291] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5748.707615] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5748.733019] LustreError: Skipped 1 previous similar message [ 5748.749255] Lustre: Skipped 1 previous similar message [ 5748.754745] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5750.758156] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5750.764336] Lustre: Skipped 2 previous similar messages [ 5751.868090] LustreError: 206706:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5752.014590] Lustre: server umount lustre-OST0001 complete [ 5755.876680] LustreError: 206066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5755.880115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.901486] LustreError: 206066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5755.927598] Lustre: Skipped 1 previous similar message [ 5755.943202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5755.945958] Lustre: Skipped 1 previous similar message [ 5760.997023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5760.997343] LustreError: 204784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5761.014317] Lustre: Skipped 1 previous similar message [ 5761.039338] LustreError: 204784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5766.121088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5766.121100] LustreError: 204784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.129960] Lustre: Skipped 1 previous similar message [ 5766.180153] LustreError: 204784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5770.209229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5770.284031] LustreError: 206909:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5770.291529] LustreError: 206909:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5770.456882] Lustre: server umount lustre-OST0000 complete [ 5774.305256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5774.313882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.332780] Lustre: Skipped 1 previous similar message [ 5774.345060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5780.430178] LustreError: 207112:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5780.445750] LustreError: 207112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5780.604285] Lustre: server umount lustre-MDT0000 complete [ 5781.475300] LustreError: 203186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5781.501541] LustreError: 203186:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5785.861874] LustreError: 203166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343074 with bad export cookie 17961006771301001965 [ 5785.865158] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5785.874700] LustreError: 203166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5786.121274] LustreError: 207314:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5786.137929] LustreError: 207314:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5786.674587] Lustre: server umount lustre-MDT0001 complete [ 5796.849037] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 5799.433444] Key type lgssc unregistered [ 5799.686338] LNet: 207847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5799.703052] LNetError: 207847:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5799.715082] LNet: Removed LNI 192.168.202.132@tcp [ 5800.509461] Key type .llcrypt unregistered [ 5800.516381] Key type ._llcrypt unregistered [ 5831.360494] Key type ._llcrypt registered [ 5831.362335] Key type .llcrypt registered [ 5831.478513] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 5844.119247] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 5845.173439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5845.208773] alg: No test for adler32 (adler32-zlib) [ 5846.282545] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 5846.502389] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 5848.176225] Key type lgssc registered [ 5849.282852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5857.191656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5863.278715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5868.810493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5874.807384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5881.178941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5881.219769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5881.235048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5882.457381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5882.512163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5882.611889] Lustre: lustre-MDT0000: new disk, initializing [ 5882.724751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5882.755594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5886.560028] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5896.406482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5896.480643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5896.545324] Lustre: 211783:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5896.593767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5896.596493] Lustre: Skipped 1 previous similar message [ 5896.646766] Lustre: lustre-MDT0001: new disk, initializing [ 5896.735796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5896.786485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5896.801654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5900.300750] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5907.059961] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5910.488904] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5916.726639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5916.812973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5917.050630] Lustre: lustre-OST0000: new disk, initializing [ 5917.062978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5917.150523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.784820] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5923.375973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5923.399589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5923.487517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5929.354836] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5932.836828] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5933.006578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5936.697624] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5936.872735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5939.178669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5939.180619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5939.196545] Lustre: Skipped 1 previous similar message [ 5939.202739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5944.294873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5944.297743] Lustre: Skipped 1 previous similar message [ 5945.155660] LustreError: 214723:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5945.491601] Lustre: server umount lustre-OST0000 complete [ 5952.997672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5953.005048] LustreError: Skipped 1 previous similar message [ 5953.008607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5953.017827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5954.531896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.650989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5959.653852] Lustre: Skipped 2 previous similar messages [ 5964.256317] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5964.407932] LustreError: 214926:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5964.419692] LustreError: 214926:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5964.583039] Lustre: server umount lustre-MDT0000 complete [ 5968.189914] LustreError: 211777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343256 with bad export cookie 3996163639953256882 [ 5968.196212] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5968.204112] LustreError: 211777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5968.363632] LustreError: 215128:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5968.381811] LustreError: 215128:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5968.570344] Lustre: server umount lustre-MDT0001 complete [ 5976.783744] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 20:34:23 (1763343263) [ 5983.862795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5984.406690] LustreError: 215800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5984.440522] LustreError: 215800:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5984.512449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5987.858388] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 5989.863406] LustreError: 215800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.954694] LustreError: 215799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5995.458634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5995.849414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5999.204285] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6003.891823] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6007.489420] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6013.904337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6014.351360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6018.977784] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6023.519413] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6031.371536] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6033.512658] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6042.465121] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6049.114556] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6059.933180] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6059.938801] Lustre: Skipped 1 previous similar message [ 6077.065271] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6077.068942] Lustre: Skipped 1 previous similar message [ 6114.581992] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6114.586485] Lustre: Skipped 3 previous similar messages [ 6184.360508] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6184.364502] Lustre: Skipped 8 previous similar messages [ 6202.853322] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6202.855812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6202.897661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6202.905248] Lustre: Skipped 1 previous similar message [ 6202.917525] Lustre: Skipped 1 previous similar message [ 6207.554160] LustreError: 219311:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6207.566588] LustreError: 219311:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6207.654627] Lustre: server umount lustre-OST0000 complete [ 6211.049157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6211.055387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.076707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6216.777355] LustreError: 219516:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6216.791364] LustreError: 219516:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6216.926571] Lustre: server umount lustre-MDT0000 complete [ 6218.215723] LustreError: 215796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.237281] LustreError: 215796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6220.408280] LustreError: 215781:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343508 with bad export cookie 3996163639953257764 [ 6220.411943] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6220.422370] LustreError: 215781:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6220.757728] Lustre: server umount lustre-MDT0001 complete [ 6227.787172] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6229.753885] Key type lgssc unregistered [ 6230.001738] LNet: 220251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6230.010330] LNetError: 220251:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6230.024362] LNet: Removed LNI 192.168.202.132@tcp [ 6230.585494] Key type .llcrypt unregistered [ 6230.588544] Key type ._llcrypt unregistered [ 6248.394598] Key type ._llcrypt registered [ 6248.395977] Key type .llcrypt registered [ 6248.563973] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6249.640725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6249.673322] alg: No test for adler32 (adler32-zlib) [ 6250.635742] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 6250.797344] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6252.440167] Key type lgssc registered [ 6253.516708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6260.438778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6260.454438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6261.805936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.789672] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6271.982813] LustreError: 221623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6272.000673] LustreError: 221623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6274.554750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6275.005394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6279.867270] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6285.618332] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6289.337653] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6296.425653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6296.817943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6301.364344] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_default_debug -1 all [ 6305.969410] Lustre: DEBUG MARKER: oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6322.660381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.662713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6322.669154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6322.686387] Lustre: Skipped 1 previous similar message [ 6323.751994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6323.761839] Lustre: Skipped 2 previous similar messages [ 6328.403113] LustreError: 224029:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6328.562891] Lustre: server umount lustre-OST0000 complete [ 6334.948144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.953064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6334.977125] Lustre: Skipped 1 previous similar message [ 6334.981878] Lustre: Skipped 1 previous similar message [ 6338.501201] LustreError: 224235:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6338.504193] LustreError: 224235:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6338.672327] Lustre: server umount lustre-MDT0000 complete [ 6340.069374] LustreError: 221624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6342.818996] LustreError: 221604:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343631 with bad export cookie 16456354501058445792 [ 6342.819549] LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6342.828025] LustreError: 221604:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6342.952982] LustreError: 224437:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6342.957341] LustreError: 224437:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6343.162689] Lustre: server umount lustre-MDT0001 complete [ 6351.656585] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6354.434888] Key type lgssc unregistered [ 6354.793108] LNet: 224968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6354.799096] LNetError: 224968:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6354.817055] LNet: Removed LNI 192.168.202.132@tcp [ 6355.675323] Key type .llcrypt unregistered [ 6355.676907] Key type ._llcrypt unregistered [ 6378.409788] Key type ._llcrypt registered [ 6378.411591] Key type .llcrypt registered [ 6378.517476] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6378.762471] Key type .llcrypt unregistered [ 6378.765633] Key type ._llcrypt unregistered [ 6404.011944] Key type ._llcrypt registered [ 6404.014170] Key type .llcrypt registered [ 6404.137229] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing set_hostid [ 6418.549285] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing load_modules_local [ 6419.440796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6419.525193] alg: No test for adler32 (adler32-zlib) [ 6420.558023] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 6420.752419] LNet: Added LNI 192.168.202.132@tcp [8/256/0/180] [ 6422.472204] Key type lgssc registered [ 6423.641284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6431.268437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.042393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6443.258073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6449.222520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6450.906152] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6228 sec ========== 20:42:17 (1763343737) [ 6452.798337] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:42:19 (1763343739) === [ 6456.687340] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:42:22 (1763343742) === [ 6501.020437] Lustre: DEBUG MARKER: oleg232-server.virtnet: executing unload_modules_local [ 6504.960901] Key type lgssc unregistered [ 6505.342724] LNet: 231082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6505.349807] LNetError: 231082:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6505.371778] LNet: Removed LNI 192.168.202.132@tcp [ 6506.298595] Key type .llcrypt unregistered [ 6506.300175] Key type ._llcrypt unregistered