[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 497844099 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003114] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.006008] kvm-guest: setup PV IPIs [ 0.008758] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011125] LSM: Security Framework initializing [ 0.012043] Yama: becoming mindful. [ 0.013000] SELinux: Initializing. [ 0.013055] *** VALIDATE selinux *** [ 0.020723] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025512] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026135] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] *** VALIDATE tmpfs *** [ 0.030374] *** VALIDATE proc *** [ 0.031185] *** VALIDATE cgroup *** [ 0.032006] *** VALIDATE cgroup2 *** [ 0.034019] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035124] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041202] debug: unmapping init [mem 0xffffffffb6059000-0xffffffffb6060fff] [ 0.043220] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044705] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046010] ... bit width: 48 [ 0.047008] ... generic registers: 4 [ 0.048007] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050009] ... fixed-purpose events: 3 [ 0.051007] ... event mask: 000000070000000f [ 0.052377] rcu: Hierarchical SRCU implementation. [ 0.054403] smp: Bringing up secondary CPUs ... [ 0.055559] x86: Booting SMP configuration: [ 0.056019] .... node #0, CPUs: #1 #2 #3 [ 0.071015] smp: Brought up 1 node, 4 CPUs [ 0.073012] smpboot: Max logical packages: 1 [ 0.074014] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.151000] node 0 deferred pages initialised in 71ms [ 0.152133] devtmpfs: initialized [ 0.153208] x86/mm: Memory block size: 128MB [ 0.155696] gcov: version magic: 0x41383552 [ 0.157149] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158066] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159215] pinctrl core: initialized pinctrl subsystem [ 0.160148] [ 0.160686] ************************************************************* [ 0.161010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162008] ** ** [ 0.163009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164008] ** ** [ 0.165012] ** This means that this kernel is built to expose internal ** [ 0.166011] ** IOMMU data structures, which may compromise security on ** [ 0.167007] ** your system. ** [ 0.168011] ** ** [ 0.169009] ** If you see this message and you are not debugging the ** [ 0.170010] ** kernel, report this immediately to your vendor! ** [ 0.171008] ** ** [ 0.172008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173008] ************************************************************* [ 0.174693] NET: Registered protocol family 16 [ 0.175433] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.177065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178803] cpuidle: using governor menu [ 0.179794] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180501] PCI: Using configuration type 1 for base access [ 0.181135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192191] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199040] cryptd: max_cpu_qlen set to 1000 [ 0.202257] ACPI: Added _OSI(Module Device) [ 0.203012] ACPI: Added _OSI(Processor Device) [ 0.204012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208508] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213146] ACPI: Interpreter enabled [ 0.214058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215012] ACPI: Using IOAPIC for interrupt routing [ 0.216109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.217390] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226201] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.227034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.229075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.231122] acpiphp: Slot [2] registered [ 0.232134] acpiphp: Slot [3] registered [ 0.233068] acpiphp: Slot [4] registered [ 0.234143] acpiphp: Slot [5] registered [ 0.237197] acpiphp: Slot [6] registered [ 0.239098] acpiphp: Slot [7] registered [ 0.240098] acpiphp: Slot [8] registered [ 0.241135] acpiphp: Slot [9] registered [ 0.244145] acpiphp: Slot [10] registered [ 0.247169] acpiphp: Slot [11] registered [ 0.249316] acpiphp: Slot [12] registered [ 0.250060] acpiphp: Slot [13] registered [ 0.251057] acpiphp: Slot [14] registered [ 0.252063] acpiphp: Slot [15] registered [ 0.253079] acpiphp: Slot [16] registered [ 0.256160] acpiphp: Slot [17] registered [ 0.257099] acpiphp: Slot [18] registered [ 0.258086] acpiphp: Slot [19] registered [ 0.259091] acpiphp: Slot [20] registered [ 0.260078] acpiphp: Slot [21] registered [ 0.262159] acpiphp: Slot [22] registered [ 0.263068] acpiphp: Slot [23] registered [ 0.264088] acpiphp: Slot [24] registered [ 0.265179] acpiphp: Slot [25] registered [ 0.266062] acpiphp: Slot [26] registered [ 0.267064] acpiphp: Slot [27] registered [ 0.268076] acpiphp: Slot [28] registered [ 0.273051] acpiphp: Slot [29] registered [ 0.274110] acpiphp: Slot [30] registered [ 0.275062] acpiphp: Slot [31] registered [ 0.276059] PCI host bridge to bus 0000:00 [ 0.278013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.280017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.281020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.283016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.285019] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.288031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.289150] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.293085] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306867] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.323062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.325023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.328016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.331032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.334745] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.337908] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.340065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.341819] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.344799] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.356023] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.361821] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.364000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.365000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.367038] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.381076] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.395637] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.409025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.421077] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.451037] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.462169] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.469021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.476052] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.488039] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.498073] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.502018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.508024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.519038] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.535292] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.551018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.557018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.575022] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.585620] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.601036] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.609021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.649020] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.665000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.667285] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.671386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.673274] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.677257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.686324] iommu: Default domain type: Passthrough [ 0.687394] SCSI subsystem initialized [ 0.689103] ACPI: bus type USB registered [ 0.695080] usbcore: registered new interface driver usbfs [ 0.699113] usbcore: registered new interface driver hub [ 0.705083] usbcore: registered new device driver usb [ 0.707123] pps_core: LinuxPPS API ver. 1 registered [ 0.708006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.714083] PTP clock support registered [ 0.716798] EDAC MC: Ver: 3.0.0 [ 0.719688] PCI: Using ACPI for IRQ routing [ 0.720725] NetLabel: Initializing [ 0.722005] NetLabel: domain hash size = 128 [ 0.723007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.725077] NetLabel: unlabeled traffic allowed by default [ 0.727262] vgaarb: loaded [ 0.729129] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.731011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.739000] clocksource: Switched to clocksource kvm-clock [ 0.883059] VFS: Disk quotas dquot_6.6.0 [ 0.885025] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.887500] *** VALIDATE ramfs *** [ 0.889155] *** VALIDATE hugetlbfs *** [ 0.891534] pnp: PnP ACPI init [ 0.893732] pnp: PnP ACPI: found 6 devices [ 0.912960] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.916776] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.918780] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.920940] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.923171] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.925438] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.928556] NET: Registered protocol family 2 [ 0.931033] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.935908] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.939038] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.944186] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.947794] TCP: Hash tables configured (established 65536 bind 65536) [ 0.951703] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.955605] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.958094] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960304] NET: Registered protocol family 1 [ 0.962660] RPC: Registered named UNIX socket transport module. [ 0.964833] RPC: Registered udp transport module. [ 0.966651] RPC: Registered tcp transport module. [ 0.968593] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.971222] NET: Registered protocol family 44 [ 0.972937] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.977598] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.979735] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.982163] PCI: CLS 0 bytes, default 64 [ 0.985204] Unpacking initramfs... [ 3.290224] debug: unmapping init [mem 0xffff9ffa3cc54000-0xffff9ffa3ffbffff] [ 3.295136] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.297910] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.301460] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.981136] Initialise system trusted keyrings [ 3.982888] Key type blacklist registered [ 3.984947] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.996861] zbud: loaded [ 4.002198] *** VALIDATE nfs *** [ 4.003095] *** VALIDATE nfs4 *** [ 4.004311] pstore: using deflate compression [ 4.010973] Platform Keyring initialized [ 4.193675] NET: Registered protocol family 38 [ 4.195490] Key type asymmetric registered [ 4.197068] Asymmetric key parser 'x509' registered [ 4.203619] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.208122] io scheduler mq-deadline registered [ 4.210268] io scheduler kyber registered [ 4.212263] io scheduler bfq registered [ 4.213890] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.217659] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.225232] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.228943] ACPI: Power Button [PWRF] [ 4.374536] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.505724] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.889623] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.073531] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.379090] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.435621] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.476358] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.481933] Non-volatile memory driver v1.3 [ 5.483870] Linux agpgart interface v0.103 [ 5.514481] virtio_blk virtio1: [vda] 133928 512-byte logical blocks (68.6 MB/65.4 MiB) [ 5.517268] vda: detected capacity change from 0 to 68571136 [ 5.543270] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.549108] vdb: detected capacity change from 0 to 1073741824 [ 5.582970] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.586100] vdc: detected capacity change from 0 to 2621440000 [ 5.612612] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.619505] vdd: detected capacity change from 0 to 2621440000 [ 5.641034] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.647059] vde: detected capacity change from 0 to 4294967296 [ 5.665870] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.672331] vdf: detected capacity change from 0 to 4294967296 [ 5.682770] libphy: Fixed MDIO Bus: probed [ 5.691649] usbcore: registered new interface driver usbserial_generic [ 5.694895] usbserial: USB Serial support registered for generic [ 5.700593] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.712881] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.717568] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.722524] mousedev: PS/2 mouse device common for all mice [ 5.724514] rtc_cmos 00:05: RTC can wake from S4 [ 5.727582] rtc_cmos 00:05: registered as rtc0 [ 5.731570] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.733524] intel_pstate: CPU model not supported [ 5.742491] hid: raw HID events driver (C) Jiri Kosina [ 5.747521] usbcore: registered new interface driver usbhid [ 5.748457] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.757880] usbhid: USB HID core driver [ 5.758088] drop_monitor: Initializing network drop monitor service [ 5.758218] Initializing XFRM netlink socket [ 5.758618] NET: Registered protocol family 10 [ 5.760080] Segment Routing with IPv6 [ 5.760125] NET: Registered protocol family 17 [ 5.760242] mpls_gso: MPLS GSO support [ 5.788344] RAS: Correctable Errors collector initialized. [ 5.794180] AVX version of gcm_enc/dec engaged. [ 5.797514] AES CTR mode by8 optimization enabled [ 5.808958] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.824363] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.991932] sched_clock: Marking stable (5991211817, 0)->(7081973472, -1090761655) [ 6.000553] registered taskstats version 1 [ 6.003088] Loading compiled-in X.509 certificates [ 6.006432] zswap: loaded using pool lzo/zbud [ 6.055324] Key type big_key registered [ 6.074152] Key type encrypted registered [ 6.075209] ima: No TPM chip found, activating TPM-bypass! [ 6.076790] ima: Allocated hash algorithm: sha1 [ 6.088044] ima: No architecture policies found [ 6.089690] evm: Initialising EVM extended attributes: [ 6.091772] evm: security.selinux [ 6.094492] evm: security.ima [ 6.095786] evm: security.capability [ 6.097771] evm: HMAC attrs: 0x1 [ 6.102842] rtc_cmos 00:05: setting system clock to 2025-12-03 03:40:01 UTC (1764733201) [ 6.118779] debug: unmapping init [mem 0xffffffffb7003000-0xffffffffb71fffff] [ 6.122222] debug: unmapping init [mem 0xffffffffb5d82000-0xffffffffb6058fff] [ 6.132103] Write protecting the kernel read-only data: 28672k [ 6.145360] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 6.151527] debug: unmapping init [mem 0xffffffffb4d14000-0xffffffffb4dfffff] [ 6.262274] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.271521] systemd[1]: Detected virtualization kvm. [ 6.273662] systemd[1]: Detected architecture x86-64. [ 6.275664] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.315522] systemd[1]: No hostname configured. [ 6.317069] systemd[1]: Set hostname to . [ 6.323103] random: systemd: uninitialized urandom read (16 bytes read) [ 6.325356] systemd[1]: Initializing machine ID from random generator. [ 6.464048] random: ln: uninitialized urandom read (6 bytes read) [ 6.677502] random: systemd: uninitialized urandom read (16 bytes read) [ 6.679834] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 6.690521] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.698252] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.248167] device-mapper: uevent: version 1.0.3 [ 8.251413] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] [ 9.775176] random: fast init done Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.948736] virtio_net virtio0 ens2: renamed from eth0 [ 10.388397] scsi host0: ata_piix [ 10.425708] scsi host1: ata_piix [ 10.446732] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.450679] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.609479] random: crng init done [ 15.611236] random: 7 urandom warning(s) missed due to ratelimiting [ 16.266862] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.884042] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.358290] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.113803] SELinux: Disabled at runtime. [ 21.220983] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.245771] systemd[1]: Detected virtualization kvm. [ 21.251585] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.020132] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.029872] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.067543] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.090365] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.094854] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.127957] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.168669] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System...[ 23.494140] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.835526] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.748456] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.777855] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.661343] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.786936] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.012466] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.609349] NFS: Registering the id_resolver key type [ 31.613352] Key type id_resolver registered [ 31.620801] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg457-server login: [ 75.160235] libcfs: loading out-of-tree module taints kernel. [ 75.194916] Key type ._llcrypt registered [ 75.198961] Key type .llcrypt registered [ 75.294898] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 93.301887] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 95.224204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.250932] alg: No test for adler32 (adler32-zlib) [ 96.936894] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 98.217685] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 99.983165] Key type lgssc registered [ 102.357885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.457791] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 118.229748] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.523434] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 130.088461] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.017133] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.083700] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 157.626865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 157.675063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 157.685099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.868135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.904701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 158.961110] Lustre: lustre-MDT0000: new disk, initializing [ 159.016043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.035820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.812468] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.236896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.292375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 171.342614] Lustre: 6518:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 171.371379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 171.373889] Lustre: Skipped 1 previous similar message [ 171.422477] Lustre: lustre-MDT0001: new disk, initializing [ 171.472389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.504396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 171.508539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.287632] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.029153] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 184.792769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 184.854295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 185.068806] Lustre: lustre-OST0000: new disk, initializing [ 185.073421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.128615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 189.479114] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.516557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.530439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.558720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.459869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.499485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 199.564982] Lustre: lustre-OST0001: new disk, initializing [ 199.567683] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 199.604311] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.351532] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.409173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.416707] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 207.463645] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 212.935785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 224.227871] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 236.200051] hrtimer: interrupt took 8548268 ns [ 237.766100] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.590868] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing yml_node [ 249.247811] Lustre: DEBUG MARKER: Client: 2.17.0.RC1 [ 252.277925] Lustre: DEBUG MARKER: MDS: 2.17.0.RC1 [ 254.918831] Lustre: DEBUG MARKER: OSS: 2.17.0.RC1 [ 257.149907] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Dec 2 22:44:11 EST 2025 [ 278.060398] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 279.688922] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 299.487701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 299.491357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.495814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 299.510537] Lustre: Skipped 2 previous similar messages [ 301.271993] LustreError: 12226:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 301.368953] Lustre: server umount lustre-MDT0000 complete [ 304.803679] LustreError: 9484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733500 with bad export cookie 1756083927805918488 [ 304.807470] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 304.826982] LustreError: 9484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 309.728541] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.729798] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.732840] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 309.732848] Lustre: Skipped 3 previous similar messages [ 309.736167] LustreError: 6525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 309.771345] Lustre: Skipped 2 previous similar messages [ 311.031123] LustreError: 12427:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 311.053481] LustreError: 12427:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 311.273168] Lustre: server umount lustre-MDT0001 complete [ 314.851167] LustreError: 12629:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 314.861992] LustreError: 12629:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 314.922629] Lustre: server umount lustre-OST0000 complete [ 318.221110] LustreError: 12830:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 318.223922] LustreError: 12830:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 318.330691] Lustre: server umount lustre-OST0001 complete [ 323.886462] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 332.212618] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 341.963298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 347.945580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 353.726094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 360.258586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 366.931281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 367.025856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.427694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 367.463616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 367.623183] Lustre: lustre-MDT0000: new disk, initializing [ 367.757892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 367.786044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 371.673884] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 382.162854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 382.242408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 382.335939] Lustre: 15016:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 382.415766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 382.419033] Lustre: Skipped 1 previous similar message [ 382.493885] Lustre: lustre-MDT0001: new disk, initializing [ 382.630611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 382.713097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 382.730773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 386.393496] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 393.529339] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 396.806449] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 403.916770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 404.019587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 404.338981] Lustre: lustre-OST0000: new disk, initializing [ 404.366405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 404.483695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.631766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 405.635674] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 405.800069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 410.777431] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 417.805586] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 422.044212] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 422.367769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 426.675982] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 426.921058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 431.585250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.591674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 431.594690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 431.594701] Lustre: Skipped 1 previous similar message [ 431.595436] Lustre: Skipped 1 previous similar message [ 436.705974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 436.712121] Lustre: Skipped 2 previous similar messages [ 441.824733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 441.841176] Lustre: Skipped 1 previous similar message [ 443.359140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 443.423113] LustreError: 17956:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 443.444453] LustreError: 17956:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 443.635865] Lustre: server umount lustre-OST0000 complete [ 449.504860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 449.509491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.064799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 452.076351] Lustre: Skipped 2 previous similar messages [ 453.944118] LustreError: 18159:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 453.947433] LustreError: 18159:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 454.086386] Lustre: server umount lustre-MDT0000 complete [ 457.190287] LustreError: 15029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 457.209127] LustreError: 15029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 457.728694] LustreError: 15010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733653 with bad export cookie 1756083927805920014 [ 457.733922] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 457.740160] LustreError: 15010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 458.092238] Lustre: server umount lustre-MDT0001 complete [ 461.420506] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 22:47:36 (1764733656) [ 468.180105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.654777] LustreError: 18886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.775391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.865283] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 474.089327] LustreError: 18885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 479.210127] LustreError: 18885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.000722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 487.208857] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 492.609788] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 496.095200] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 503.554888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 510.143325] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 515.480687] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 522.211065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.223902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.228036] Lustre: Skipped 1 previous similar message [ 522.232548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.771525] LustreError: 21131:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 527.774134] LustreError: 21131:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 527.941527] Lustre: server umount lustre-OST0000 complete [ 533.474859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 533.484065] LustreError: Skipped 1 previous similar message [ 533.487480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.493928] Lustre: Skipped 1 previous similar message [ 546.271311] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 546.517891] Lustre: server umount lustre-MDT0000 complete [ 548.834825] LustreError: 19598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.850785] LustreError: 19598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 550.307942] LustreError: 18867:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733745 with bad export cookie 1756083927805920903 [ 550.319439] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.644972] Lustre: server umount lustre-MDT0001 complete [ 562.293633] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 565.914579] Key type lgssc unregistered [ 566.301547] LNet: 22068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.316366] LNetError: 22068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.337797] LNet: Removed LNI 192.168.204.157@tcp [ 568.537189] Key type .llcrypt unregistered [ 568.538989] Key type ._llcrypt unregistered [ 592.866066] Key type ._llcrypt registered [ 592.867566] Key type .llcrypt registered [ 593.018664] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 594.077446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 594.217137] alg: No test for adler32 (adler32-zlib) [ 595.383272] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 595.662654] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 597.375143] Key type lgssc registered [ 598.454671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 606.634822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 606.680210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.345797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 612.564678] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 618.467085] LustreError: 23464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.494721] LustreError: 23464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 621.961630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 622.417803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 626.564025] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 631.953252] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 636.067198] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 644.232880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 644.716681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.606155] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 652.969498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 656.328343] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 672.740386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 672.744301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 672.790415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 673.773732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.778237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 674.498132] LustreError: 26157:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 674.599464] Lustre: server umount lustre-OST0000 complete [ 678.884508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.886323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 678.890429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 678.916925] Lustre: Skipped 1 previous similar message [ 678.949366] Lustre: Skipped 2 previous similar messages [ 684.000525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 684.004040] Lustre: Skipped 1 previous similar message [ 684.550156] LustreError: 26358:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 684.560275] LustreError: 26358:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 684.722094] Lustre: server umount lustre-MDT0000 complete [ 688.612351] LustreError: 23444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733884 with bad export cookie 8807082491777021588 [ 688.616781] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 688.632847] LustreError: 23444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 688.806881] LustreError: 26559:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 688.816837] LustreError: 26559:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 689.046243] Lustre: server umount lustre-MDT0001 complete [ 700.215429] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 702.978951] Key type lgssc unregistered [ 703.330362] LNet: 27091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.347367] LNetError: 27091:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.357287] LNet: Removed LNI 192.168.204.157@tcp [ 705.362891] Key type .llcrypt unregistered [ 705.364671] Key type ._llcrypt unregistered [ 727.188235] Key type ._llcrypt registered [ 727.191728] Key type .llcrypt registered [ 727.286252] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 728.252201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 728.277791] alg: No test for adler32 (adler32-zlib) [ 729.316180] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 729.521774] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 731.215993] Key type lgssc registered [ 732.597387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 743.333559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 743.362696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.856237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.232327] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 755.172311] LustreError: 28489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 755.178923] LustreError: 28489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 759.832484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 760.287914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 760.437266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 764.306937] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 769.408624] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 772.376074] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 783.034890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 783.273801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 788.384585] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 790.451336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 793.589409] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 800.737027] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 800.749983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.773284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 800.779787] Lustre: Skipped 1 previous similar message [ 805.818173] LustreError: 31080:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 805.936497] Lustre: server umount lustre-OST0000 complete [ 810.977126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 810.980292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 810.988292] Lustre: Skipped 2 previous similar messages [ 815.604353] LustreError: 31282:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 815.607548] LustreError: 31282:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 815.698403] Lustre: server umount lustre-MDT0000 complete [ 816.099739] LustreError: 31081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 816.123285] LustreError: 31081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 819.111135] LustreError: 28469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734014 with bad export cookie 18056056389794300199 [ 819.132987] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 819.307689] LustreError: 31485:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 819.318161] LustreError: 31485:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 819.503887] Lustre: server umount lustre-MDT0001 complete [ 828.204512] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 830.629544] Key type lgssc unregistered [ 830.923968] LNet: 32017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.938203] LNetError: 32017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.952539] LNet: Removed LNI 192.168.204.157@tcp [ 831.672221] Key type .llcrypt unregistered [ 831.674507] Key type ._llcrypt unregistered [ 850.046266] Key type ._llcrypt registered [ 850.047948] Key type .llcrypt registered [ 850.166163] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 850.772142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.781179] alg: No test for adler32 (adler32-zlib) [ 851.819119] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 852.050958] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 853.711179] Key type lgssc registered [ 854.740291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 861.513151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 861.532782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.969619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.090701] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 873.444635] LustreError: 33389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.458811] LustreError: 33389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 873.485056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 873.849927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 877.036621] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 881.601552] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 884.834583] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 890.802775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 891.190889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 895.852512] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 900.065789] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 908.588643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 919.009654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.016923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 919.028858] Lustre: Skipped 1 previous similar message [ 919.042200] Lustre: Skipped 1 previous similar message [ 922.426344] LustreError: 35660:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 922.578783] Lustre: server umount lustre-OST0000 complete [ 929.248950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.255456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 929.264620] Lustre: Skipped 1 previous similar message [ 929.271391] Lustre: Skipped 1 previous similar message [ 932.078529] LustreError: 35862:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 932.086347] LustreError: 35862:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 932.199346] Lustre: server umount lustre-MDT0000 complete [ 934.368647] LustreError: 33390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.392071] LustreError: 33390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 935.617440] LustreError: 33370:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734131 with bad export cookie 17222126541083098855 [ 935.624103] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 935.635350] LustreError: 33370:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 935.762582] LustreError: 36063:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 935.777364] LustreError: 36063:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 936.027750] Lustre: server umount lustre-MDT0001 complete [ 945.659315] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 947.831380] Key type lgssc unregistered [ 948.102791] LNet: 36596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.106933] LNetError: 36596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.161101] LNet: Removed LNI 192.168.204.157@tcp [ 949.792495] Key type .llcrypt unregistered [ 949.795308] Key type ._llcrypt unregistered [ 982.268482] Key type ._llcrypt registered [ 982.271616] Key type .llcrypt registered [ 982.419650] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 985.199474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 985.207491] alg: No test for adler32 (adler32-zlib) [ 986.492327] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 987.067458] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 988.879829] Key type lgssc registered [ 991.201888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1003.046034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1003.067225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.851074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1008.627905] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1015.268212] LustreError: 37990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1015.293516] LustreError: 37990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1016.299900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1016.695510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1020.238140] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1025.719102] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1029.544520] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1036.450182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1036.962821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1042.623076] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1047.509710] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1050.294620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1058.342943] Lustre: lustre-OST0000: Not available for connect from 192.168.204.57@tcp (stopping) [ 1060.835091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1060.838151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.843736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1060.852844] Lustre: Skipped 1 previous similar message [ 1063.031945] LustreError: 40239:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1063.224442] Lustre: server umount lustre-OST0000 complete [ 1081.313634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.317486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1081.330692] Lustre: Skipped 1 previous similar message [ 1081.339929] Lustre: Skipped 1 previous similar message [ 1086.709944] LustreError: 40613:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1086.729104] LustreError: 40613:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1086.876686] Lustre: server umount lustre-MDT0000 complete [ 1087.463228] LustreError: 37991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1087.488488] LustreError: 37991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1090.561674] LustreError: 37971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734285 with bad export cookie 6308152520824797042 [ 1090.581508] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1096.740511] LustreError: 40816:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1096.752850] LustreError: 40816:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1096.996339] Lustre: server umount lustre-MDT0001 complete [ 1105.838376] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1108.150057] Key type lgssc unregistered [ 1108.383551] LNet: 41349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.391141] LNetError: 41349:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.407859] LNet: Removed LNI 192.168.204.157@tcp [ 1109.119220] Key type .llcrypt unregistered [ 1109.121173] Key type ._llcrypt unregistered [ 1127.236989] Key type ._llcrypt registered [ 1127.238736] Key type .llcrypt registered [ 1127.348780] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1128.047191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1128.072371] alg: No test for adler32 (adler32-zlib) [ 1129.071909] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1129.246780] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1130.911182] Key type lgssc registered [ 1131.763786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1137.813905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1137.834207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1139.262109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1142.794112] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1149.415184] LustreError: 42719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.424960] LustreError: 42719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1150.065087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1150.377906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1153.447887] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1158.322569] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1161.293716] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1165.959902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1166.231540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.558894] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1174.976918] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1183.669849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1191.397148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1191.400698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.412328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.987622] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.57@tcp (stopping) [ 1193.180091] LustreError: 44988:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1193.231752] Lustre: server umount lustre-MDT0000 complete [ 1193.956536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.963952] LustreError: 42715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.971445] LustreError: 42715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1196.150299] LustreError: 42700:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734391 with bad export cookie 13337032422032100193 [ 1196.160136] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1197.064705] LustreError: 42715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.081184] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.57@tcp (stopping) [ 1198.316492] LustreError: 45188:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1198.322065] LustreError: 45188:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1198.448862] Lustre: server umount lustre-MDT0001 complete [ 1210.204058] LustreError: 42700:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.57@tcp arrived at 1764734405 with bad export cookie 13337032422032100858 [ 1210.217878] LustreError: 42700:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1228.621338] LustreError: 45412:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1228.625798] LustreError: 45412:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1228.740749] Lustre: server umount lustre-OST0000 complete [ 1242.373102] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1244.518736] Key type lgssc unregistered [ 1244.754601] LNet: 46252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1244.760494] LNetError: 46252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1244.782577] LNet: Removed LNI 192.168.204.157@tcp [ 1245.518886] Key type .llcrypt unregistered [ 1245.524147] Key type ._llcrypt unregistered [ 1264.843651] Key type ._llcrypt registered [ 1264.845930] Key type .llcrypt registered [ 1264.949060] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1265.809390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1265.935064] alg: No test for adler32 (adler32-zlib) [ 1266.998840] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1267.200106] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1268.847188] Key type lgssc registered [ 1269.895409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1276.004568] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1276.031199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1291.551514] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1296.671230] LustreError: 47610:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 1323.359187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1327.417628] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1331.073124] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1497.626997] LustreError: 48317:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 1497.704249] Lustre: server umount lustre-OST0000 complete [ 1507.630973] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1509.142199] Key type lgssc unregistered [ 1509.289447] LNet: 49153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.296712] LNetError: 49153:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.308146] LNet: Removed LNI 192.168.204.157@tcp [ 1509.687436] Key type .llcrypt unregistered [ 1509.690482] Key type ._llcrypt unregistered [ 1522.136025] Key type ._llcrypt registered [ 1522.137447] Key type .llcrypt registered [ 1522.194866] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1522.685829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1522.697194] alg: No test for adler32 (adler32-zlib) [ 1523.620212] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1523.738575] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1525.335194] Key type lgssc registered [ 1525.974723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1529.871847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1529.881295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.146715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1532.967431] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1537.162144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1537.387151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1539.348462] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1542.415738] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1544.332316] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1547.794669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1548.013942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.860443] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1553.433720] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1554.595125] LustreError: 52771:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1554.650233] Lustre: server umount lustre-OST0000 complete [ 1557.985813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1557.990714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1557.998227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1561.058536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1561.059255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.633133] LustreError: 52972:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1562.636236] LustreError: 52972:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1562.692460] Lustre: server umount lustre-MDT0000 complete [ 1563.104508] LustreError: 50525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1563.112091] LustreError: 50525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1564.654215] LustreError: 52147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734760 with bad export cookie 2440475278997675211 [ 1564.659066] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1564.662714] LustreError: 52147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1564.741731] LustreError: 53174:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1564.743742] LustreError: 53174:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1564.813577] Lustre: server umount lustre-MDT0001 complete [ 1569.581332] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1570.829708] Key type lgssc unregistered [ 1570.981192] LNet: 53707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.988963] LNetError: 53707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.996705] LNet: Removed LNI 192.168.204.157@tcp [ 1571.394069] Key type .llcrypt unregistered [ 1571.395766] Key type ._llcrypt unregistered [ 1582.973799] Key type ._llcrypt registered [ 1582.975377] Key type .llcrypt registered [ 1583.037795] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1583.624391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1583.635785] alg: No test for adler32 (adler32-zlib) [ 1584.520755] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1584.634053] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1586.251339] Key type lgssc registered [ 1586.892658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1590.532023] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1590.545252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1606.047352] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1611.167207] LustreError: 55049:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 1637.791205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.358088] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1642.683862] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1645.676591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.924415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1645.957785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1647.714770] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1651.170338] LustreError: 55076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.177279] LustreError: 55076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1651.476416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1651.660861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1653.314042] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1655.677392] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1657.373733] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1661.408499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1661.412186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.422635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1661.920547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1661.924270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.930121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1662.948698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1662.951742] Lustre: Skipped 1 previous similar message [ 1664.310879] LustreError: 57326:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1664.379394] Lustre: server umount lustre-OST0000 complete [ 1681.464990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.465954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.470800] Lustre: Skipped 1 previous similar message [ 1686.497370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1686.500426] Lustre: Skipped 2 previous similar messages [ 1687.626512] LustreError: 57704:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1687.629341] LustreError: 57704:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1687.666849] Lustre: server umount lustre-MDT0000 complete [ 1688.546258] LustreError: 55974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1688.553646] LustreError: 55974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1689.768795] LustreError: 55057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734885 with bad export cookie 18313147398294068477 [ 1689.769924] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1689.776288] LustreError: 55057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1695.885132] LustreError: 57905:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1695.888440] LustreError: 57905:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1695.962430] Lustre: server umount lustre-MDT0001 complete [ 1701.004650] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1702.364548] Key type lgssc unregistered [ 1702.530045] LNet: 58437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.533955] LNetError: 58437:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.546557] LNet: Removed LNI 192.168.204.157@tcp [ 1702.951688] Key type .llcrypt unregistered [ 1702.953135] Key type ._llcrypt unregistered [ 1714.472692] Key type ._llcrypt registered [ 1714.474263] Key type .llcrypt registered [ 1714.520271] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1715.005214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1715.016437] alg: No test for adler32 (adler32-zlib) [ 1715.925924] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1716.045901] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1717.656758] Key type lgssc registered [ 1718.203951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1722.195073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1722.203597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.433982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.514418] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1729.849149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1730.055882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1732.006606] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1734.529893] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1736.191858] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1739.397129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1739.576175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.228166] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1744.624025] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1747.757889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1761.251366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1761.254518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.261217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1763.298968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1763.300268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1763.305069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.718163] LustreError: 62072:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1764.768769] Lustre: server umount lustre-OST0000 complete [ 1768.416985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.418153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1768.421318] Lustre: Skipped 1 previous similar message [ 1768.424911] Lustre: Skipped 2 previous similar messages [ 1772.554911] LustreError: 62274:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1772.558512] LustreError: 62274:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1772.605368] Lustre: server umount lustre-MDT0000 complete [ 1773.536152] LustreError: 59807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1773.543391] LustreError: 59807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1774.460915] LustreError: 59789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734969 with bad export cookie 4559114902775590214 [ 1774.465515] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1774.472163] LustreError: 59789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1774.544941] LustreError: 62475:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1774.547214] LustreError: 62475:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1774.626369] Lustre: server umount lustre-MDT0001 complete [ 1779.516332] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1780.827659] Key type lgssc unregistered [ 1780.982044] LNet: 63007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.985809] LNetError: 63007:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.000521] LNet: Removed LNI 192.168.204.157@tcp [ 1781.459271] Key type .llcrypt unregistered [ 1781.461033] Key type ._llcrypt unregistered [ 1798.652352] Key type ._llcrypt registered [ 1798.654314] Key type .llcrypt registered [ 1798.702193] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1799.104201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1799.153395] alg: No test for adler32 (adler32-zlib) [ 1800.034130] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1800.131266] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1801.727185] Key type lgssc registered [ 1802.204630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1805.545936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1805.556265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1806.759538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.373843] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1812.408415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1812.651222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1814.528811] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1817.040119] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1818.679200] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1821.773323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1821.861256] LustreError: 66270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1821.869187] LustreError: 66270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1821.937908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.320850] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1826.554123] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1832.234518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1839.492852] Lustre: Failing over lustre-MDT0000 [ 1839.655565] LustreError: 66939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1839.698575] Lustre: server umount lustre-MDT0000 complete [ 1842.199951] LustreError: 66279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1842.657986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.667196] Lustre: Skipped 2 previous similar messages [ 1843.880873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.951295] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1844.017161] Lustre: *** cfs_fail_loc=135, val=0*** [ 1844.063120] LustreError: 67450:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 1844.067527] LustreError: 67450:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.157@tcp: cfg command failed: rc = -2 [ 1844.071958] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1844.071958] [ 1844.078424] LustreError: MGC192.168.204.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1844.085544] LustreError: 67438:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 1844.090179] LustreError: 67438:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 1844.095788] LustreError: 67438:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1844.110253] Lustre: server umount lustre-MDT0000 complete [ 1844.112673] LustreError: 67438:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 1847.043230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1847.111371] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1847.244377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.285087] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1847.303889] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1848.895704] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1852.389432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1852.405137] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1852.424357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1854.944870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1854.949818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.956432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1857.506879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1857.508621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.508868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.950160] LustreError: 68257:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1859.955146] LustreError: 68257:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1860.015857] Lustre: server umount lustre-OST0000 complete [ 1862.624849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1862.627857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.630082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.634318] Lustre: Skipped 1 previous similar message [ 1862.636877] Lustre: Skipped 1 previous similar message [ 1867.744914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1867.747477] Lustre: Skipped 1 previous similar message [ 1867.911451] LustreError: 68458:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1867.914157] LustreError: 68458:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1867.963213] Lustre: server umount lustre-MDT0000 complete [ 1869.632812] LustreError: 66941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735065 with bad export cookie 15625130576909593673 [ 1869.636393] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1869.639031] LustreError: 66941:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1869.774943] Lustre: server umount lustre-MDT0001 complete [ 1874.445400] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1875.688674] Key type lgssc unregistered [ 1875.836618] LNet: 69190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.839347] LNetError: 69190:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.849068] LNet: Removed LNI 192.168.204.157@tcp [ 1876.218274] Key type .llcrypt unregistered [ 1876.220146] Key type ._llcrypt unregistered [ 1887.871717] Key type ._llcrypt registered [ 1887.873173] Key type .llcrypt registered [ 1887.922532] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1888.373519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1888.406195] alg: No test for adler32 (adler32-zlib) [ 1889.296055] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1889.398956] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1890.999178] Key type lgssc registered [ 1891.514204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.836130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1894.848166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.068226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.854678] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1901.810591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1901.978407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1903.621880] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1905.872090] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1907.538481] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1910.714448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1910.884810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1913.346532] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1915.719920] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1921.129620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1928.766235] Lustre: Failing over lustre-MDT0000 [ 1928.883459] LustreError: 72824:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1928.921279] Lustre: server umount lustre-MDT0000 complete [ 1931.289737] LustreError: 70557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.301178] LustreError: 70557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1931.744982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.750812] Lustre: Skipped 2 previous similar messages [ 1933.545479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.611968] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.680069] Lustre: *** cfs_fail_loc=a05, val=0*** [ 1933.742649] LustreError: 73333:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-9) [ 1933.745076] LustreError: 73333:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.157@tcp: cfg command failed: rc = -9 [ 1933.748667] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 1933.748667] [ 1933.753545] LustreError: MGC192.168.204.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 1933.758548] LustreError: 73321:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 1933.761140] LustreError: 73321:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -9 [ 1933.763780] LustreError: 73321:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 1933.769713] LustreError: 70541:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735129 with bad export cookie 2334282165590721838 [ 1933.773807] Lustre: server umount lustre-MDT0000 complete [ 1933.778797] LustreError: 70541:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1933.789154] LustreError: 73321:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 1936.390370] LustreError: 70557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1936.399208] LustreError: 70557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1936.777428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1938.912673] LustreError: 70561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.922129] LustreError: 70561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1941.511862] LustreError: 70555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.038092] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x20650943159085c8 [ 1944.044190] Lustre: MGC192.168.204.157@tcp: Connection restored to 0@lo (at 0@lo) [ 1944.169281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.193331] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1945.775182] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1946.629899] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1949.670139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1949.681354] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1949.710125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1953.248050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1953.252272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.258812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1954.788029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1954.789157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1954.795871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.235216] LustreError: 74149:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1957.249956] LustreError: 74149:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1957.301478] Lustre: server umount lustre-OST0000 complete [ 1959.906951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1959.907550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1959.908280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.908284] Lustre: Skipped 1 previous similar message [ 1959.922108] Lustre: Skipped 1 previous similar message [ 1965.027178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1965.029889] Lustre: Skipped 2 previous similar messages [ 1965.135231] LustreError: 74350:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1965.137886] LustreError: 74350:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1965.171934] Lustre: server umount lustre-MDT0000 complete [ 1966.754343] LustreError: 70540:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735162 with bad export cookie 2334282165590721992 [ 1966.757035] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1966.758562] LustreError: 70540:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1966.766553] LustreError: Skipped 1 previous similar message [ 1966.894369] Lustre: server umount lustre-MDT0001 complete [ 1970.844460] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1972.011255] Key type lgssc unregistered [ 1972.148406] LNet: 75082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.151634] LNetError: 75082:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.160354] LNet: Removed LNI 192.168.204.157@tcp [ 1972.485802] Key type .llcrypt unregistered [ 1972.487088] Key type ._llcrypt unregistered [ 1977.357178] Key type ._llcrypt registered [ 1977.358672] Key type .llcrypt registered [ 1977.409288] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 1977.536334] Key type .llcrypt unregistered [ 1977.538076] Key type ._llcrypt unregistered [ 1987.490757] Key type ._llcrypt registered [ 1987.492179] Key type .llcrypt registered [ 1987.535732] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 1987.971744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1987.985265] alg: No test for adler32 (adler32-zlib) [ 1988.852168] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1988.954484] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 1990.551218] Key type lgssc registered [ 1991.026981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1994.025381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1994.034952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.236702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1996.673350] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 1999.823694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1999.972751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2001.325239] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2003.301502] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2004.649097] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2007.290382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2007.450818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.769789] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2011.931136] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2019.752795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2025.952594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2025.956439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.963065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2030.049353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2030.051030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2030.051630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.053045] Lustre: Skipped 1 previous similar message [ 2031.470102] LustreError: 79187:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2031.518313] Lustre: server umount lustre-OST0000 complete [ 2035.170277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.170888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2035.175615] Lustre: Skipped 1 previous similar message [ 2035.180825] Lustre: Skipped 1 previous similar message [ 2039.110984] LustreError: 79388:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2039.114051] LustreError: 79388:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2039.148242] Lustre: server umount lustre-MDT0000 complete [ 2040.288525] LustreError: 76921:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2040.294993] LustreError: 76921:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2040.603308] LustreError: 79389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735236 with bad export cookie 16004431507651770036 [ 2040.606837] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2040.607597] LustreError: 79389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2040.683292] LustreError: 79590:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2040.686196] LustreError: 79590:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2040.751223] Lustre: server umount lustre-MDT0001 complete [ 2044.836360] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2046.043129] Key type lgssc unregistered [ 2046.185381] LNet: 80121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.188227] LNetError: 80121:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.197312] LNet: Removed LNI 192.168.204.157@tcp [ 2046.540129] Key type .llcrypt unregistered [ 2046.541843] Key type ._llcrypt unregistered [ 2056.283431] Key type ._llcrypt registered [ 2056.284887] Key type .llcrypt registered [ 2056.332282] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2056.702026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.743435] alg: No test for adler32 (adler32-zlib) [ 2057.609361] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2057.700446] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2059.287209] Key type lgssc registered [ 2059.731838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2062.531424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2062.541358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.742107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.321375] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2068.602279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2068.742217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.245470] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2072.538424] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2074.040455] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2076.754304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2076.897565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.277057] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2081.580195] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2089.191760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2094.561221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2094.563789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.564559] LustreError: Skipped 1 previous similar message [ 2094.571120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2094.574635] Lustre: Skipped 1 previous similar message [ 2099.682795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2099.685201] Lustre: Skipped 1 previous similar message [ 2100.656091] LustreError: 83763:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2100.703567] Lustre: server umount lustre-OST0000 complete [ 2104.802162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.803037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.806434] Lustre: Skipped 2 previous similar messages [ 2104.811622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2108.351358] LustreError: 83965:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2108.354389] LustreError: 83965:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2108.381186] Lustre: server umount lustre-MDT0000 complete [ 2109.849497] LustreError: 81475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735305 with bad export cookie 9755474703562176822 [ 2109.852298] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.854262] LustreError: 81475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2109.928605] LustreError: 84169:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2109.930721] LustreError: 84169:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2109.985478] Lustre: server umount lustre-MDT0001 complete [ 2113.970331] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2115.105804] Key type lgssc unregistered [ 2115.243654] LNet: 84699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.247378] LNetError: 84699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.258515] LNet: Removed LNI 192.168.204.157@tcp [ 2115.597625] Key type .llcrypt unregistered [ 2115.599481] Key type ._llcrypt unregistered [ 2125.198291] Key type ._llcrypt registered [ 2125.200044] Key type .llcrypt registered [ 2125.249312] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2125.571890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2125.601049] alg: No test for adler32 (adler32-zlib) [ 2126.477149] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2126.574938] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2128.167492] Key type lgssc registered [ 2128.643512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.171227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.181251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.497877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.849777] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2142.700539] LustreError: 86069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.734953] LustreError: 86069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2146.586484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2147.039242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2150.710645] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2155.366759] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2158.292469] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2164.262971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2164.567920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.995664] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2175.387040] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2177.902521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2188.259135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.259968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2188.269271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2188.276233] Lustre: Skipped 1 previous similar message [ 2193.376588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2193.380482] Lustre: Skipped 1 previous similar message [ 2193.727202] LustreError: 88338:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2193.812303] Lustre: server umount lustre-OST0000 complete [ 2198.498205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2198.500209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.504744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2198.508061] Lustre: Skipped 2 previous similar messages [ 2203.622198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2203.625269] Lustre: Skipped 1 previous similar message [ 2208.737247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.741867] Lustre: Skipped 1 previous similar message [ 2210.271152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2210.319855] LustreError: 88539:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2210.322675] LustreError: 88539:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2210.357963] Lustre: server umount lustre-MDT0000 complete [ 2212.314544] LustreError: 86049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735407 with bad export cookie 5904226845435892017 [ 2212.320880] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2212.417551] LustreError: 88742:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2212.420218] LustreError: 88742:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2212.512313] Lustre: server umount lustre-MDT0001 complete [ 2217.847540] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2219.355364] Key type lgssc unregistered [ 2219.519949] LNet: 89274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.526696] LNetError: 89274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.536837] LNet: Removed LNI 192.168.204.157@tcp [ 2220.017335] Key type .llcrypt unregistered [ 2220.019415] Key type ._llcrypt unregistered [ 2233.683691] Key type ._llcrypt registered [ 2233.685285] Key type .llcrypt registered [ 2233.775300] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2234.469690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2234.486893] alg: No test for adler32 (adler32-zlib) [ 2235.394701] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2235.562615] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2237.183169] Key type lgssc registered [ 2237.784684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2242.204511] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2242.220750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2257.695329] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2262.815645] LustreError: 90618:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 2289.503149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.858340] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2294.421728] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2297.407519] LustreError: 91483:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2297.459682] Lustre: server umount lustre-OST0000 complete [ 2301.746843] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 23:18:16 (1764735496) [ 2304.857362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.082259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2307.210630] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2310.113410] LustreError: 92154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.120680] LustreError: 92154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2311.238123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2311.466975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2313.322205] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2315.919109] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2317.988852] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2321.507628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2321.719033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.570903] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2327.402954] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2329.901154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 2345.441254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2345.444710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2345.453268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2346.471992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.472423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2346.473185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.157114] LustreError: 94576:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2348.160065] LustreError: 94576:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2348.216865] Lustre: server umount lustre-OST0000 complete [ 2351.587935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.591867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.595977] Lustre: Skipped 1 previous similar message [ 2351.602309] Lustre: Skipped 2 previous similar messages [ 2356.258855] LustreError: 94777:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2356.262189] LustreError: 94777:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2356.313873] Lustre: server umount lustre-MDT0000 complete [ 2356.707125] LustreError: 92155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.719090] LustreError: 92155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2358.420329] LustreError: 92135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735553 with bad export cookie 1347774169317284243 [ 2358.424231] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2358.436628] LustreError: 92135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2358.526058] LustreError: 94978:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2358.529071] LustreError: 94978:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2358.619445] Lustre: server umount lustre-MDT0001 complete [ 2363.470552] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2364.752845] Key type lgssc unregistered [ 2364.912434] LNet: 95510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.916591] LNetError: 95510:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2364.927508] LNet: Removed LNI 192.168.204.157@tcp [ 2365.289880] Key type .llcrypt unregistered [ 2365.291407] Key type ._llcrypt unregistered [ 2375.825817] Key type ._llcrypt registered [ 2375.827476] Key type .llcrypt registered [ 2375.875290] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2376.225156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2376.258064] alg: No test for adler32 (adler32-zlib) [ 2377.146097] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2377.262295] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2378.855233] Key type lgssc registered [ 2379.356496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2382.821474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2382.831127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.056073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.547218] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2388.818776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2388.957864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2390.461988] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2392.498922] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2393.861696] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2396.398993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2396.539615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.748701] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2400.651475] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2408.810576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 2414.050102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2414.050164] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2414.051649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2414.053803] Lustre: Skipped 1 previous similar message [ 2415.789132] LustreError: 99150:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2415.828726] Lustre: server umount lustre-OST0000 complete [ 2419.168958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.170937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.174226] Lustre: Skipped 1 previous similar message [ 2419.177157] Lustre: Skipped 1 previous similar message [ 2419.330764] LustreError: 99353:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2419.333158] LustreError: 99353:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2419.363273] Lustre: server umount lustre-MDT0000 complete [ 2420.724318] LustreError: 99354:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735616 with bad export cookie 15115097861460301486 [ 2420.727175] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2420.728519] LustreError: 99354:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2420.799379] LustreError: 99554:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2420.801777] LustreError: 99554:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2420.857700] Lustre: server umount lustre-MDT0001 complete [ 2424.921801] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2426.061415] Key type lgssc unregistered [ 2426.213548] LNet: 100085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.217375] LNetError: 100085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.228483] LNet: Removed LNI 192.168.204.157@tcp [ 2426.551208] Key type .llcrypt unregistered [ 2426.552967] Key type ._llcrypt unregistered [ 2434.155182] Key type ._llcrypt registered [ 2434.156865] Key type .llcrypt registered [ 2434.196821] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2434.563422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2434.571905] alg: No test for adler32 (adler32-zlib) [ 2435.458970] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2435.562404] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2437.161783] Key type lgssc registered [ 2437.612396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2440.415503] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2440.423567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2455.839317] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2460.959184] LustreError: 101285:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 2487.583426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.044601] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2492.396955] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2495.057335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.195390] LustreError: MGC192.168.204.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2495.200470] LustreError: 102196:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2495.203976] LustreError: 102196:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 2495.207982] LustreError: 102196:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 2495.246542] Lustre: server umount lustre-MDT0000 complete [ 2495.248468] LustreError: 102196:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2507.743650] LustreError: 102627:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2507.785180] Lustre: server umount lustre-OST0000 complete [ 2511.303601] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 2515.236426] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2518.873198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2521.312837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2523.577541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2526.125350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2528.822930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2528.849077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.945960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2528.961556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2529.002923] Lustre: lustre-MDT0000: new disk, initializing [ 2529.037336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.047394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2530.581208] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2535.278081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2535.308119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2535.346385] Lustre: 104960:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2535.373948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2535.376300] Lustre: Skipped 1 previous similar message [ 2535.416579] Lustre: lustre-MDT0001: new disk, initializing [ 2535.459734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2535.476236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2535.479931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2537.021283] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2540.582577] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2542.018699] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2544.681899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2544.716124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2544.830700] Lustre: lustre-OST0000: new disk, initializing [ 2544.833883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2544.872100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.174957] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2550.720931] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2552.353804] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2554.875365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2554.879578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2554.914478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2555.504150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2557.148972] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2557.252032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2559.970045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2559.970102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.971468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2559.973633] LustreError: Skipped 1 previous similar message [ 2559.978456] Lustre: Skipped 1 previous similar message [ 2564.082075] LustreError: 107923:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2564.085021] LustreError: 107923:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2564.120938] Lustre: server umount lustre-OST0000 complete [ 2566.111810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2566.115329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.120578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2566.122844] Lustre: Skipped 1 previous similar message [ 2570.208208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2570.208679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.588711] LustreError: 108124:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2571.591214] LustreError: 108124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2571.626169] Lustre: server umount lustre-MDT0000 complete [ 2573.009210] LustreError: 105742:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735768 with bad export cookie 2295329593306283589 [ 2573.012437] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2573.014668] LustreError: 105742:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2573.136227] Lustre: server umount lustre-MDT0001 complete [ 2576.163440] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 23:22:51 (1764735771) [ 2577.374874] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 2583.058886] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 2586.911247] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2590.597643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2592.947757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2595.140931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2597.446458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2599.952765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2599.984259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.086248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2600.101143] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2600.148370] Lustre: lustre-MDT0000: new disk, initializing [ 2600.179964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.186811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2601.690758] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2606.240362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2606.277067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2606.313903] Lustre: 111455:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2606.338631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2606.340822] Lustre: Skipped 1 previous similar message [ 2606.380274] Lustre: lustre-MDT0001: new disk, initializing [ 2606.436120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2606.439672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2607.910969] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2611.239218] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2612.731172] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2615.226208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2615.255507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2615.357466] Lustre: lustre-OST0000: new disk, initializing [ 2615.360323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2615.391097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2615.393557] Lustre: Skipped 1 previous similar message [ 2617.603754] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2617.664680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2617.668992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2617.727316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2621.058909] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2622.533721] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2622.610069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2624.024460] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2624.099142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2626.798091] LustreError: 114393:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2626.800665] LustreError: 114393:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2626.840331] Lustre: server umount lustre-OST0000 complete [ 2632.162349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2632.165965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.171565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2632.173765] Lustre: Skipped 1 previous similar message [ 2638.304213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2638.307013] Lustre: Skipped 3 previous similar messages [ 2638.308714] LustreError: 114594:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2638.311469] LustreError: 114594:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2638.342028] Lustre: server umount lustre-MDT0000 complete [ 2639.770260] LustreError: 112236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735835 with bad export cookie 2295329593306284457 [ 2639.773323] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2639.774987] LustreError: 112236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2650.084451] LustreError: 114799:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2650.086528] LustreError: 114799:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2650.147684] Lustre: server umount lustre-MDT0001 complete [ 2653.198165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.396233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.796746] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2658.015502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2659.501593] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2661.433809] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2662.748491] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2665.317462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2667.721278] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2670.149829] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2684.296869] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 2687.969499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2687.970307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2687.975197] Lustre: Skipped 1 previous similar message [ 2687.976989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2687.978886] Lustre: Skipped 1 previous similar message [ 2691.311087] LustreError: 117637:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2691.313080] LustreError: 117637:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2691.352305] Lustre: server umount lustre-OST0000 complete [ 2698.775501] Lustre: server umount lustre-MDT0000 complete [ 2700.027085] LustreError: 115257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735895 with bad export cookie 2295329593306285339 [ 2700.030440] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.030638] LustreError: 115257:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2700.147691] Lustre: server umount lustre-MDT0001 complete [ 2703.921118] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 2705.058555] Key type lgssc unregistered [ 2705.203497] LNet: 118572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2705.207439] LNetError: 118572:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2705.214447] LNet: Removed LNI 192.168.204.157@tcp [ 2705.539647] Key type .llcrypt unregistered [ 2705.541137] Key type ._llcrypt unregistered [ 2711.735573] Key type ._llcrypt registered [ 2711.736849] Key type .llcrypt registered [ 2711.781564] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 2717.916219] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 2718.222272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2718.231432] alg: No test for adler32 (adler32-zlib) [ 2719.084957] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 2719.169477] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 2720.751198] Key type lgssc registered [ 2721.143382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2724.063495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2726.221839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2728.346529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2730.736337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2733.436932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2733.458437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2733.464468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.562375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2734.574878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2734.619280] Lustre: lustre-MDT0000: new disk, initializing [ 2734.653194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.660709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2736.151363] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2740.396360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2740.421374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2740.460320] Lustre: 121673:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2740.497825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2740.500309] Lustre: Skipped 1 previous similar message [ 2740.538776] Lustre: lustre-MDT0001: new disk, initializing [ 2740.573030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2740.587461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2740.590819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2741.914117] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2745.094740] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2746.513769] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2748.914122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2748.937696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2749.040422] Lustre: lustre-OST0000: new disk, initializing [ 2749.042955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2749.074857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.119857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2749.125929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2749.162702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2751.164555] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2754.356098] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2755.773898] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2755.851517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2757.224821] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2757.309759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2759.649641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2759.650174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.652020] LustreError: Skipped 1 previous similar message [ 2759.657404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.660680] Lustre: Skipped 1 previous similar message [ 2764.015103] LustreError: 124614:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2764.061798] Lustre: server umount lustre-OST0000 complete [ 2766.304307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2766.307053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2766.310932] Lustre: Skipped 1 previous similar message [ 2766.313553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2769.888881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2769.889203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.459592] LustreError: 124815:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2771.461553] LustreError: 124815:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2771.497052] Lustre: server umount lustre-MDT0000 complete [ 2772.657405] LustreError: 122469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735968 with bad export cookie 8867992434925010172 [ 2772.661216] LustreError: 122469:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2772.661268] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.729752] LustreError: 125017:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2772.733291] LustreError: 125017:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2772.796540] Lustre: server umount lustre-MDT0001 complete [ 2775.687772] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 23:26:11 (1764735971) [ 2777.950724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.090067] LustreError: 125687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.097364] LustreError: 125687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2778.115610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.465290] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2782.395448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2782.531447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2783.889720] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2785.721112] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2786.931285] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2787.609238] LustreError: 125717:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2787.611207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.611710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2787.611715] Lustre: Skipped 1 previous similar message [ 2787.614382] LustreError: 125717:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2787.618012] Lustre: Skipped 1 previous similar message [ 2787.625257] LustreError: 125717:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 2791.712705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.715322] Lustre: Skipped 2 previous similar messages [ 2793.724275] LustreError: 127027:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2793.726544] LustreError: 127027:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2793.751828] Lustre: server umount lustre-MDT0000 complete [ 2794.976188] LustreError: 125668:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735990 with bad export cookie 8867992434925011026 [ 2794.979387] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2794.980023] LustreError: 125668:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2795.099246] Lustre: server umount lustre-MDT0001 complete [ 2797.770952] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 23:26:33 (1764735993) [ 2799.814163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2815.263426] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2820.383380] LustreError: 127874:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 2845.983457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2847.922232] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2849.846600] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2850.534907] LustreError: 128537:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 2850.536867] LustreError: 128537:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2850.571521] Lustre: server umount lustre-OST0000 complete [ 2853.294910] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 23:27:28 (1764736048) [ 2855.566441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2855.759614] LustreError: 129207:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.210255] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2860.155672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2861.746823] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2863.711795] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2864.800513] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2866.885830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2866.984802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.987471] Lustre: Skipped 2 previous similar messages [ 2868.783863] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2870.468762] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2887.648751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2887.650207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.653501] Lustre: Skipped 1 previous similar message [ 2887.658216] Lustre: Skipped 2 previous similar messages [ 2893.757982] LustreError: 131474:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2893.760492] LustreError: 131474:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2893.787942] Lustre: server umount lustre-MDT0000 complete [ 2895.022021] LustreError: 130827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736090 with bad export cookie 8867992434925011656 [ 2895.024809] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2895.027202] LustreError: 130827:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2895.131662] Lustre: server umount lustre-MDT0001 complete [ 2906.524707] Lustre: server umount lustre-OST0000 complete [ 2909.065164] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 23:28:24 (1764736104) [ 2911.210042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.348533] LustreError: 132547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.371179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.684653] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2915.354148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2916.710935] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2918.382787] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2919.493535] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2921.624867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2923.549993] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 2925.235069] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2926.495390] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2936.103726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 2936.772159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2938.055591] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2938.122740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2940.383854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2940.387387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.392938] Lustre: Skipped 1 previous similar message [ 2940.395338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2940.398095] Lustre: Skipped 3 previous similar messages [ 2944.815680] LustreError: 135360:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2944.817697] LustreError: 135360:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2944.862889] Lustre: server umount lustre-OST0000 complete [ 2946.528903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2946.531843] LustreError: Skipped 1 previous similar message [ 2952.277984] Lustre: server umount lustre-MDT0000 complete [ 2953.637234] LustreError: 132528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736149 with bad export cookie 8867992434925012713 [ 2953.639659] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.642152] LustreError: 132528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2953.777455] Lustre: server umount lustre-MDT0001 complete [ 2956.674937] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 23:29:12 (1764736152) [ 2959.285209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2974.752782] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2979.871476] LustreError: 136409:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 3005.599250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.602191] Lustre: Skipped 2 previous similar messages [ 3011.396960] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3017.726731] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3024.528309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.008755] LustreError: 136435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.133572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 3028.310745] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3030.502616] LustreError: 136435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3033.529197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3035.903896] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3039.242540] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3041.340424] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3043.786326] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3043.917472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3046.109400] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3046.277494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3049.446181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3049.451922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3049.460840] Lustre: Skipped 3 previous similar messages [ 3049.471112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3049.474823] Lustre: Skipped 6 previous similar messages [ 3053.643657] LustreError: 139183:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3053.646714] LustreError: 139183:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3053.699161] Lustre: server umount lustre-MDT0000 complete [ 3055.589962] LustreError: 136436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.602354] LustreError: 136436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3055.644052] LustreError: 136417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736251 with bad export cookie 8867992434925013721 [ 3055.646401] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3055.649948] LustreError: 136417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3055.822964] Lustre: server umount lustre-MDT0001 complete [ 3068.484317] Lustre: server umount lustre-OST0000 complete [ 3072.690800] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 23:31:07 (1764736267) [ 3076.307722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3091.809665] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3096.927447] LustreError: 140233:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 3125.708478] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3128.444980] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3131.823332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.017565] LustreError: 140260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.102157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 3133.958461] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3137.606898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3137.770373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3137.773155] Lustre: Skipped 4 previous similar messages [ 3139.405436] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3142.262757] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3144.132559] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3148.167388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3148.213112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3148.310788] Lustre: lustre-OST0001: new disk, initializing [ 3148.313710] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3151.186453] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3154.715746] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3156.306693] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3157.504971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3157.509708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3157.556941] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3158.451559] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3160.399639] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3160.516488] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3162.601945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.607643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3162.615191] Lustre: Skipped 3 previous similar messages [ 3162.617442] Lustre: Skipped 3 previous similar messages [ 3167.546091] LustreError: 143975:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3167.548233] LustreError: 143975:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3167.608971] Lustre: server umount lustre-OST0000 complete [ 3167.717279] LustreError: 141891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3167.728584] LustreError: 141891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3172.832561] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3172.839022] LustreError: Skipped 1 previous similar message [ 3176.084825] Lustre: server umount lustre-OST0001 complete [ 3184.642345] Lustre: server umount lustre-MDT0000 complete [ 3186.695141] LustreError: 140240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736382 with bad export cookie 8867992434925014666 [ 3186.697659] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3186.706585] LustreError: 140240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3186.919649] Lustre: server umount lustre-MDT0001 complete [ 3190.395583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3193.086796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3195.814166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3198.716957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3203.123339] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 23:33:18 (1764736398) [ 3204.303725] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 3205.438422] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 3206.539726] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 23:33:21 (1764736401) [ 3211.269672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3211.478974] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3211.505287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3214.261867] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3220.102830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3220.177371] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3220.194393] Lustre: 146216:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3223.061738] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3226.599830] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3228.867776] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3233.415071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3233.536678] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3233.903857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 3237.367556] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3241.144157] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3243.938755] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3244.100371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3246.750435] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3246.892839] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3249.120703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3249.142138] LustreError: Skipped 1 previous similar message [ 3249.144257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.156902] Lustre: Skipped 4 previous similar messages [ 3254.478496] Lustre: server umount lustre-OST0000 complete [ 3272.426980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3276.262126] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3279.872377] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3283.060744] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3290.026479] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3290.039918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3290.043899] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3290.520323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3293.166905] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3293.318271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3295.668707] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffffa0c706f73800.ost_server_uuid 50 [ 3296.616926] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa0c706f73800.ost_server_uuid in FULL state after 0 sec [ 3296.756906] LustreError: 147355:0:(ldlm_lockd.c:726:ldlm_handle_ast_error()) ### client (nid 192.168.204.57@tcp) returned error from blocking AST (req@00000000ab80e062 x1850459725956864 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000c299646c/0x7b1170228a8e97dd lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.57@tcp remote: 0x8a813b7de4c4b9e8 expref: 5 pid: 146224 timeout: 3396 lvb_type: 0 [ 3296.773160] LustreError: lustre-MDT0000: A client on nid 192.168.204.57@tcp was evicted due to a lock blocking callback time out: rc -107 [ 3296.779910] LustreError: 146212:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.57@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000c299646c/0x7b1170228a8e97dd lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.57@tcp remote: 0x8a813b7de4c4b9e8 expref: 6 pid: 146224 timeout: 0 lvb_type: 0 [ 3300.329689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3300.331267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3300.333709] LustreError: Skipped 2 previous similar messages [ 3300.336763] Lustre: Skipped 9 previous similar messages [ 3305.014123] LustreError: 150814:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3305.026260] LustreError: 150814:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3305.112501] Lustre: server umount lustre-OST0000 complete [ 3313.755703] Lustre: server umount lustre-MDT0000 complete [ 3315.680481] LustreError: 146224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.690862] LustreError: 146224:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3315.931329] LustreError: 146944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736511 with bad export cookie 8867992434925015842 [ 3315.933668] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3315.956402] LustreError: 146944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3322.183299] Lustre: server umount lustre-MDT0001 complete [ 3329.250525] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3331.132647] Key type lgssc unregistered [ 3331.328457] LNet: 151749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3331.333225] LNetError: 151749:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3331.345220] LNet: Removed LNI 192.168.204.157@tcp [ 3331.888990] Key type .llcrypt unregistered [ 3331.891123] Key type ._llcrypt unregistered [ 3346.507951] Key type ._llcrypt registered [ 3346.509574] Key type .llcrypt registered [ 3346.603864] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3347.233171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3347.256950] alg: No test for adler32 (adler32-zlib) [ 3348.212613] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 3348.367461] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3349.983271] Key type lgssc registered [ 3350.884698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3356.170737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3356.190681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.474356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.842399] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3364.786551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3364.992164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.381796] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3370.818557] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3373.064246] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3377.590179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3377.821598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.382971] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3381.937778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3384.490923] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3392.229770] Lustre: Failing over lustre-MDT0000 [ 3392.360792] LustreError: 155369:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3392.420709] Lustre: server umount lustre-MDT0000 complete [ 3395.553768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.555355] LustreError: 153121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3395.566970] Lustre: Skipped 2 previous similar messages [ 3395.586608] LustreError: 153121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3400.674182] LustreError: 153124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3400.686499] LustreError: 153124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3404.052539] LustreError: 153106:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.57@tcp arrived at 1764736599 with bad export cookie 11240904001190166879 [ 3404.067263] LustreError: 153106:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3405.798701] LustreError: 153124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3405.810062] LustreError: 153124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3409.013740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.105292] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3409.261432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.303260] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3411.913790] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3413.331134] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3413.340583] Lustre: lustre-MDT0000: Denying connection for new client 670af7d8-ee79-4708-bbae-b4f2f92e63c4 (at 192.168.204.57@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 3414.503179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3421.154762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3421.164598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.180458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.742375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.743192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.005587] LustreError: 156303:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3427.012457] LustreError: 156303:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3427.084599] Lustre: server umount lustre-OST0000 complete [ 3429.685492] LustreError: 156504:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3429.697671] Lustre: 155869:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3429.707403] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3429.720665] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3429.734187] LustreError: 155869:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000001b0e8b7 x1850460385350144/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3429.747101] LustreError: 155869:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 3429.752973] LustreError: 155869:0:(fid_request.c:316:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 3429.759392] Lustre: lustre-MDT0000: Recovery over after 0:16, of 2 clients 0 recovered and 2 were evicted. [ 3429.866238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.873080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3429.875978] Lustre: Skipped 1 previous similar message [ 3429.906453] LustreError: 156504:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3429.910510] LustreError: 156504:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3429.990237] Lustre: server umount lustre-MDT0000 complete [ 3432.757192] LustreError: 153105:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736628 with bad export cookie 11240904001190166984 [ 3432.758432] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3432.767394] LustreError: 153105:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3432.777268] LustreError: 153124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.979200] LustreError: 156705:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3438.991356] LustreError: 156705:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3439.150527] Lustre: server umount lustre-MDT0001 complete [ 3446.403168] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3448.501545] Key type lgssc unregistered [ 3448.720496] LNet: 157237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3448.732955] LNetError: 157237:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3448.741992] LNet: Removed LNI 192.168.204.157@tcp [ 3449.312453] Key type .llcrypt unregistered [ 3449.316942] Key type ._llcrypt unregistered [ 3465.213994] Key type ._llcrypt registered [ 3465.218884] Key type .llcrypt registered [ 3465.317820] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3465.852389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3465.877152] alg: No test for adler32 (adler32-zlib) [ 3466.841731] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 3467.000748] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3468.631175] Key type lgssc registered [ 3469.396159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3474.230131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3474.241860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3475.509648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3477.573604] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3482.975610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3483.199373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3485.673663] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3488.403584] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3490.337581] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3494.480437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3494.686252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.755314] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3499.817884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 3500.413769] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3505.122579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3505.128905] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3505.129593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3506.744181] LustreError: 160852:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3506.792682] Lustre: server umount lustre-OST0000 complete [ 3508.703983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3508.707816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.716221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3513.314097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.316069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.837819] LustreError: 161053:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3514.842207] LustreError: 161053:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3514.911295] Lustre: server umount lustre-MDT0000 complete [ 3517.250147] LustreError: 158589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736712 with bad export cookie 16927660635501797757 [ 3517.255187] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3517.256145] LustreError: 158589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3517.364372] LustreError: 161254:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3517.374135] LustreError: 161254:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3517.504332] Lustre: server umount lustre-MDT0001 complete [ 3523.600160] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3526.395503] Key type lgssc unregistered [ 3526.824552] LNet: 161786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3526.834595] LNetError: 161786:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3526.860194] LNet: Removed LNI 192.168.204.157@tcp [ 3527.729266] Key type .llcrypt unregistered [ 3527.739823] Key type ._llcrypt unregistered [ 3565.601915] Key type ._llcrypt registered [ 3565.606700] Key type .llcrypt registered [ 3565.728491] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3566.790389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3566.806108] alg: No test for adler32 (adler32-zlib) [ 3568.237643] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 3568.614937] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3570.319728] Key type lgssc registered [ 3571.758701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3579.567852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3579.587778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.182508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.461238] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3590.624168] LustreError: 163471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3590.640898] LustreError: 163471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3591.227911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3591.503038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3595.147601] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3599.766459] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3602.470928] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3607.604652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3607.960314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.422525] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3616.057449] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3626.428292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 3631.586714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.587256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3631.599208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3631.603265] Lustre: Skipped 1 previous similar message [ 3632.689482] LustreError: 165739:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3632.765075] Lustre: server umount lustre-OST0000 complete [ 3636.704541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.710157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3636.715236] Lustre: Skipped 2 previous similar messages [ 3641.356842] LustreError: 165939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3641.360910] LustreError: 165939:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3641.446636] Lustre: server umount lustre-MDT0000 complete [ 3641.827301] LustreError: 163472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3641.851987] LustreError: 163472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3644.404478] LustreError: 165088:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736839 with bad export cookie 10574575634984963736 [ 3644.409358] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3644.412854] LustreError: 165088:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3644.529314] LustreError: 166141:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3644.531441] LustreError: 166141:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3644.645846] Lustre: server umount lustre-MDT0001 complete [ 3651.698964] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3653.490693] Key type lgssc unregistered [ 3653.699183] LNet: 166672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3653.707763] LNetError: 166672:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3653.728694] LNet: Removed LNI 192.168.204.157@tcp [ 3654.367793] Key type .llcrypt unregistered [ 3654.369706] Key type ._llcrypt unregistered [ 3669.964263] Key type ._llcrypt registered [ 3669.965985] Key type .llcrypt registered [ 3670.079489] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3670.617634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3670.644971] alg: No test for adler32 (adler32-zlib) [ 3671.760455] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 3671.952271] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3673.607246] Key type lgssc registered [ 3674.429260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3682.762754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3682.776988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.009371] Lustre: *** cfs_fail_loc=135, val=0*** [ 3684.068301] LustreError: 168154:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000 failed (-2) [ 3684.071806] LustreError: 168154:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.204.157@tcp: cfg command failed: rc = -2 [ 3684.083578] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3684.083578] [ 3684.090800] LustreError: MGC192.168.204.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3684.105645] LustreError: 168114:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3684.110819] LustreError: 168114:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 3684.117366] LustreError: 168114:0:(obd_config.c:836:class_cleanup()) Device 5 not setup [ 3684.133717] LustreError: 168114:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3684.233527] Lustre: server umount lustre-MDT0000 complete [ 3684.235455] LustreError: 168114:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3688.843990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.112680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.804136] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3697.660780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3697.899333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3700.113593] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3703.151567] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3705.465540] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3706.902080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3706.903713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3706.907967] Lustre: Skipped 1 previous similar message [ 3707.937273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3707.949528] Lustre: Skipped 2 previous similar messages [ 3712.995352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3712.999363] Lustre: Skipped 1 previous similar message [ 3713.107032] LustreError: 169872:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3713.109040] LustreError: 169872:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3713.169554] Lustre: server umount lustre-MDT0000 complete [ 3715.472013] LustreError: 168510:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736910 with bad export cookie 5759385699083443156 [ 3715.475294] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3715.477572] LustreError: 168510:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3715.582735] LustreError: 170074:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3715.585588] LustreError: 170074:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3715.679715] Lustre: server umount lustre-MDT0001 complete [ 3722.143719] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3724.215612] Key type lgssc unregistered [ 3724.469303] LNet: 170604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3724.480448] LNetError: 170604:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3724.497124] LNet: Removed LNI 192.168.204.157@tcp [ 3725.074863] Key type .llcrypt unregistered [ 3725.076822] Key type ._llcrypt unregistered [ 3738.172704] Key type ._llcrypt registered [ 3738.174180] Key type .llcrypt registered [ 3738.236567] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3738.361303] Key type .llcrypt unregistered [ 3738.364216] Key type ._llcrypt unregistered [ 3750.241232] Key type ._llcrypt registered [ 3750.242906] Key type .llcrypt registered [ 3750.335260] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3751.145050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3751.172964] alg: No test for adler32 (adler32-zlib) [ 3752.175876] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 3752.365288] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3754.023142] Key type lgssc registered [ 3755.110572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3761.867707] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3761.888283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3777.439365] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3782.559298] LustreError: 172672:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.157@tcp: failed processing log, type 4: rc = -110 [ 3809.183531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3812.610286] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3815.568564] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3819.303300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.583270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.620463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 3822.357452] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3828.707609] LustreError: 173595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3828.718156] LustreError: 173595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3830.228289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3830.542024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3834.215229] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3838.249954] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3841.119951] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3843.724961] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 3855.336540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.340600] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.365358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3856.353619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3856.355976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3856.360768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3856.374791] Lustre: Skipped 1 previous similar message [ 3857.496349] LustreError: 175299:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3857.644833] Lustre: server umount lustre-OST0000 complete [ 3861.474756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3861.475586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.483193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3861.488737] Lustre: Skipped 1 previous similar message [ 3861.500235] Lustre: Skipped 1 previous similar message [ 3866.596946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3866.602675] Lustre: Skipped 1 previous similar message [ 3871.712983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3871.717170] Lustre: Skipped 1 previous similar message [ 3875.295155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3875.377460] LustreError: 175500:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3875.380420] LustreError: 175500:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3875.429116] Lustre: server umount lustre-MDT0000 complete [ 3875.811747] LustreError: 173595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3875.823573] LustreError: 173595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3878.129617] LustreError: 172679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737073 with bad export cookie 4940868787547685413 [ 3878.132810] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3878.137234] LustreError: 172679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3878.266601] LustreError: 175705:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3878.271873] LustreError: 175705:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3878.449933] Lustre: server umount lustre-MDT0001 complete [ 3884.857327] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 3886.723281] Key type lgssc unregistered [ 3886.929925] LNet: 176236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3886.934416] LNetError: 176236:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3887.977656] LNet: Removed LNI 192.168.204.157@tcp [ 3888.508538] Key type .llcrypt unregistered [ 3888.510721] Key type ._llcrypt unregistered [ 3902.111647] Key type ._llcrypt registered [ 3902.113389] Key type .llcrypt registered [ 3902.212083] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 3902.843667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3902.932116] alg: No test for adler32 (adler32-zlib) [ 3903.864043] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 3904.029906] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 3905.647185] Key type lgssc registered [ 3906.332909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3911.728521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3911.744450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3913.138093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3916.155264] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3923.043253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3923.388988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3926.182297] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3929.947996] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3932.241739] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3937.107639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3937.338156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3939.378557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 3941.096154] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3944.461509] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3948.577360] Lustre: Failing over lustre-MDT0000 [ 3948.577702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.602294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3948.680042] LustreError: 179905:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3948.756115] Lustre: server umount lustre-MDT0000 complete [ 3949.024124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3949.029448] LustreError: 177607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.039379] LustreError: 177607:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3952.145454] LustreError: 178398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.634665] LustreError: 177604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3953.642022] LustreError: 177604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3957.254274] LustreError: 177604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3962.374766] LustreError: 177604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3962.385204] LustreError: 177604:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3963.710189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.785797] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.878659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 3963.881771] Lustre: Skipped 1 previous similar message [ 3963.980512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.035305] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3966.414618] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 3967.502500] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3968.962253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 3969.005507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3969.024260] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3969.052367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 3974.224119] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 3984.867683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3984.872483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.879698] Lustre: Skipped 2 previous similar messages [ 3984.883847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.888710] Lustre: Skipped 2 previous similar messages [ 3987.939863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3987.943311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.949716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3990.510080] LustreError: 181191:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3990.512243] LustreError: 181191:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3990.557436] Lustre: server umount lustre-OST0000 complete [ 3994.592553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3994.592863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.597423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3994.602588] Lustre: Skipped 1 previous similar message [ 3994.609242] Lustre: Skipped 3 previous similar messages [ 3998.363658] LustreError: 181393:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3998.371122] LustreError: 181393:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3998.449909] Lustre: server umount lustre-MDT0000 complete [ 3999.712442] LustreError: 177608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3999.719613] LustreError: 177608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4000.178303] LustreError: 177589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737195 with bad export cookie 6804450079165109307 [ 4000.180069] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4000.184380] LustreError: 177589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4000.335552] Lustre: server umount lustre-MDT0001 complete [ 4005.170054] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4006.659583] Key type lgssc unregistered [ 4006.824449] LNet: 182125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4006.831467] LNetError: 182125:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4006.838486] LNet: Removed LNI 192.168.204.157@tcp [ 4007.282447] Key type .llcrypt unregistered [ 4007.284077] Key type ._llcrypt unregistered [ 4020.043257] Key type ._llcrypt registered [ 4020.044979] Key type .llcrypt registered [ 4020.103539] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4020.621398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4020.647743] alg: No test for adler32 (adler32-zlib) [ 4021.559165] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4021.680800] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4023.295231] Key type lgssc registered [ 4023.915519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4027.801656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4027.809937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4029.050591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4031.031418] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4035.295798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4035.503644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4037.498525] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4040.324136] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4042.340727] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4045.890636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4046.081280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4048.720863] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4051.472166] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4054.317911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 4059.851565] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 4066.644236] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 4074.610037] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4085.216045] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4085.223180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.231259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4086.243502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4086.245630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.245756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4088.629302] LustreError: 185918:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4088.688633] Lustre: server umount lustre-OST0000 complete [ 4091.362116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.368022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4091.368106] Lustre: Skipped 1 previous similar message [ 4091.370348] Lustre: Skipped 1 previous similar message [ 4096.480187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4096.484199] Lustre: Skipped 1 previous similar message [ 4101.601238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4101.604850] Lustre: Skipped 1 previous similar message [ 4105.183170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4105.230732] LustreError: 186119:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4105.233765] LustreError: 186119:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4105.278471] Lustre: server umount lustre-MDT0000 complete [ 4105.701178] LustreError: 183496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4105.706615] LustreError: 183496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4107.146463] LustreError: 185700:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737302 with bad export cookie 4125350810222457224 [ 4107.147118] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4107.156443] LustreError: 185700:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4107.234511] LustreError: 186321:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4107.238425] LustreError: 186321:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4107.324839] Lustre: server umount lustre-MDT0001 complete [ 4112.109945] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4113.550768] Key type lgssc unregistered [ 4113.705558] LNet: 186852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4113.710569] LNetError: 186852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4113.719692] LNet: Removed LNI 192.168.204.157@tcp [ 4114.143872] Key type .llcrypt unregistered [ 4114.145637] Key type ._llcrypt unregistered [ 4125.165947] Key type ._llcrypt registered [ 4125.167353] Key type .llcrypt registered [ 4125.217622] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4125.600960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4125.656655] alg: No test for adler32 (adler32-zlib) [ 4126.528857] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4126.624638] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4128.215185] Key type lgssc registered [ 4128.673440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4131.884184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4131.894417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4133.110667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4134.616066] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4138.100726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4138.279835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4139.947759] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4142.253724] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4143.761498] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4146.625728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4146.765092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4149.162573] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4151.626885] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4158.054878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 4163.770210] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 4173.611979] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 4183.993652] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 4193.681159] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 4204.515183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4204.517113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4204.520085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4204.527297] Lustre: Skipped 1 previous similar message [ 4204.531287] Lustre: Skipped 1 previous similar message [ 4209.638205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4214.754049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4214.756081] Lustre: Skipped 2 previous similar messages [ 4218.335274] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4218.360143] LustreError: 192556:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4218.419568] Lustre: server umount lustre-OST0000 complete [ 4220.385095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4220.388609] LustreError: Skipped 1 previous similar message [ 4220.390813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4220.396819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4224.993078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4224.993686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4226.303045] LustreError: 192758:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4226.306060] LustreError: 192758:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4226.335862] Lustre: server umount lustre-MDT0000 complete [ 4227.854365] LustreError: 189845:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737423 with bad export cookie 8990089833904014548 [ 4227.858181] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4227.859628] LustreError: 189845:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4227.930421] LustreError: 192959:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4227.933378] LustreError: 192959:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4228.017617] Lustre: server umount lustre-MDT0001 complete [ 4232.335746] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4233.562198] Key type lgssc unregistered [ 4233.724529] LNet: 193491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4233.727894] LNetError: 193491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4233.736426] LNet: Removed LNI 192.168.204.157@tcp [ 4234.102406] Key type .llcrypt unregistered [ 4234.103758] Key type ._llcrypt unregistered [ 4244.409129] Key type ._llcrypt registered [ 4244.410695] Key type .llcrypt registered [ 4244.465405] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4244.877439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4244.885990] alg: No test for adler32 (adler32-zlib) [ 4245.774258] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4245.886425] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4247.487195] Key type lgssc registered [ 4247.976840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4251.284803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4251.292628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.498986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.154744] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4257.748876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4257.915821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4259.453614] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4261.703366] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4263.228975] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4265.921611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4266.062511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.318552] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4270.435859] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4277.353838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 4287.970108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4287.971638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4287.975911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4287.978318] Lustre: Skipped 1 previous similar message [ 4287.983507] Lustre: Skipped 1 previous similar message [ 4291.122192] LustreError: 197278:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4291.176285] Lustre: server umount lustre-OST0000 complete [ 4293.090168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.090849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4293.095530] Lustre: Skipped 1 previous similar message [ 4298.208599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4298.211450] Lustre: Skipped 2 previous similar messages [ 4298.947937] LustreError: 197479:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4298.951547] LustreError: 197479:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4298.986490] Lustre: server umount lustre-MDT0000 complete [ 4300.579338] LustreError: 194842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737495 with bad export cookie 12717151213529050221 [ 4300.582610] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4300.584339] LustreError: 194842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4300.667072] LustreError: 197681:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4300.669085] LustreError: 197681:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4300.737879] Lustre: server umount lustre-MDT0001 complete [ 4305.038780] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4306.228641] Key type lgssc unregistered [ 4306.386467] LNet: 198213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4306.390160] LNetError: 198213:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4306.401304] LNet: Removed LNI 192.168.204.157@tcp [ 4306.750414] Key type .llcrypt unregistered [ 4306.751977] Key type ._llcrypt unregistered [ 4315.557231] Key type ._llcrypt registered [ 4315.558909] Key type .llcrypt registered [ 4315.607654] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4315.977952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4316.025468] alg: No test for adler32 (adler32-zlib) [ 4316.911532] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4317.011963] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4318.607265] Key type lgssc registered [ 4319.089093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4322.679307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4322.690040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.947470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.725372] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4329.662178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4329.876720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4331.510467] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4333.888141] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4335.364749] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4338.118727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4338.319798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4340.740800] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4343.119441] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4349.610223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 4360.160101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4360.163665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.166252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4360.168588] Lustre: Skipped 1 previous similar message [ 4360.172952] Lustre: Skipped 1 previous similar message [ 4365.042153] LustreError: 201856:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4365.088949] Lustre: server umount lustre-OST0000 complete [ 4370.401196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.408251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4370.410815] Lustre: Skipped 1 previous similar message [ 4375.520741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4375.523643] Lustre: Skipped 1 previous similar message [ 4379.616416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4379.618858] Lustre: Skipped 1 previous similar message [ 4381.151149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4381.190456] LustreError: 202058:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4381.192409] LustreError: 202058:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4381.216993] Lustre: server umount lustre-MDT0000 complete [ 4382.525228] LustreError: 199417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737577 with bad export cookie 7456724553760650792 [ 4382.528139] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4382.529643] LustreError: 199417:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4382.599640] LustreError: 202260:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4382.602583] LustreError: 202260:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4382.663337] Lustre: server umount lustre-MDT0001 complete [ 4386.522611] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4387.729464] Key type lgssc unregistered [ 4387.887502] LNet: 202791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4387.890942] LNetError: 202791:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4387.903665] LNet: Removed LNI 192.168.204.157@tcp [ 4388.253054] Key type .llcrypt unregistered [ 4388.254591] Key type ._llcrypt unregistered [ 4396.656796] Key type ._llcrypt registered [ 4396.658365] Key type .llcrypt registered [ 4396.703888] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4397.070524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4397.122437] alg: No test for adler32 (adler32-zlib) [ 4398.000265] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4398.099419] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4399.695226] Key type lgssc registered [ 4400.217102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4403.424816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4403.434550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4404.622538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4406.219639] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4409.664896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4409.872313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4411.378030] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4413.509738] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4414.978855] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4417.845228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4418.005757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4420.426151] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4422.641270] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4431.337085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 4436.671824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4436.729783] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4436.847220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4439.459768] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4441.980317] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4444.900430] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 4455.550364] Lustre: Permanently deactivating lustre-OST0001 [ 4455.557314] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4462.603743] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4466.177108] Lustre: Permanently reactivating lustre-OST0001 [ 4466.187089] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 4466.190516] Lustre: Skipped 2 previous similar messages [ 4469.449641] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.456309] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4469.459981] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4469.465575] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4469.483965] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4479.463676] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.465130] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4479.470027] Lustre: Skipped 2 previous similar messages [ 4482.797149] LustreError: 207484:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4482.811559] Lustre: server umount lustre-OST0001 complete [ 4484.578860] LustreError: 205608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.579473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4484.580665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4484.580670] Lustre: Skipped 1 previous similar message [ 4484.584519] LustreError: 205608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4484.597594] Lustre: Skipped 1 previous similar message [ 4489.698827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4489.699407] LustreError: 205611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.702495] Lustre: Skipped 2 previous similar messages [ 4489.709744] LustreError: 205611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4490.612438] LustreError: 207685:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4490.617925] LustreError: 207685:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4490.680465] Lustre: server umount lustre-OST0000 complete [ 4494.817096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.817961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4494.822167] Lustre: Skipped 1 previous similar message [ 4494.826771] Lustre: Skipped 1 previous similar message [ 4498.632515] LustreError: 207886:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4498.634887] LustreError: 207886:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4498.666884] Lustre: server umount lustre-MDT0000 complete [ 4499.936577] LustreError: 204014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4499.942040] LustreError: 204014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4500.129278] LustreError: 203996:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737695 with bad export cookie 289308792550014469 [ 4500.132286] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4500.133376] LustreError: 203996:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4500.261063] Lustre: server umount lustre-MDT0001 complete [ 4504.182607] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4505.387880] Key type lgssc unregistered [ 4505.537810] LNet: 208620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4505.544655] LNetError: 208620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4505.554401] LNet: Removed LNI 192.168.204.157@tcp [ 4505.884271] Key type .llcrypt unregistered [ 4505.885738] Key type ._llcrypt unregistered [ 4521.609735] Key type ._llcrypt registered [ 4521.611112] Key type .llcrypt registered [ 4521.662868] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 4528.756553] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4529.216795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4529.233827] alg: No test for adler32 (adler32-zlib) [ 4530.104435] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4530.205193] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4531.799253] Key type lgssc registered [ 4532.300234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4536.226980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4539.000466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4541.811658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4544.574572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4547.160936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4547.205558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4547.214272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4548.316393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4548.332698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4548.375224] Lustre: lustre-MDT0000: new disk, initializing [ 4548.415815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4548.424678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4549.944913] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4554.841826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4554.867302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4554.914535] Lustre: 212538:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4554.948410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4554.951152] Lustre: Skipped 1 previous similar message [ 4554.991283] Lustre: lustre-MDT0001: new disk, initializing [ 4555.034781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4555.052876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4555.056854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4556.775485] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4560.651481] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4562.223511] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4565.186313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4565.231861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4565.366903] Lustre: lustre-OST0000: new disk, initializing [ 4565.369356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4565.410507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4567.825630] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4571.575255] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4573.239831] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4575.238175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4575.242589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4575.289298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4575.382673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4577.047905] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4577.143383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4580.322927] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4580.324991] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.326059] LustreError: Skipped 1 previous similar message [ 4580.327689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4580.331403] Lustre: Skipped 1 previous similar message [ 4583.987364] LustreError: 215489:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4584.035237] Lustre: server umount lustre-OST0000 complete [ 4585.951904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4585.954703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.960209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4585.962426] Lustre: Skipped 1 previous similar message [ 4590.560118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4590.560613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.548521] LustreError: 215690:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4591.550513] LustreError: 215690:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4591.578477] Lustre: server umount lustre-MDT0000 complete [ 4592.922460] LustreError: 213319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737788 with bad export cookie 16386694734829422474 [ 4592.925688] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4592.927994] LustreError: 213319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4593.006177] LustreError: 215893:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4593.008987] LustreError: 215893:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4593.066635] Lustre: server umount lustre-MDT0001 complete [ 4596.274438] Lustre: DEBUG MARKER: == conf-sanity test 30a: Big config llog and permanent parameter deletion ========================================================== 23:56:31 (1764737791) [ 4599.304420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.526481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4601.091739] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4604.896625] LustreError: 216562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.902423] LustreError: 216562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4604.909628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4605.088244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4606.659878] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4608.810201] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4610.261178] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4612.965731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4613.101288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4615.314736] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4617.366078] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4628.606666] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4636.183629] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 4641.706744] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 4648.309201] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4664.457144] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4664.459852] Lustre: Skipped 1 previous similar message [ 4688.215180] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 4688.217936] Lustre: Skipped 2 previous similar messages [ 4726.280094] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 4726.282079] Lustre: Skipped 4 previous similar messages [ 4793.316298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4793.316578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.320332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4793.324693] Lustre: Skipped 1 previous similar message [ 4793.329199] Lustre: Skipped 2 previous similar messages [ 4795.822089] LustreError: 220059:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4795.824655] LustreError: 220059:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4795.862368] Lustre: server umount lustre-OST0000 complete [ 4798.432939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4798.438065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4798.440824] Lustre: Skipped 1 previous similar message [ 4803.263263] LustreError: 220264:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4803.266456] LustreError: 220264:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4803.296306] Lustre: server umount lustre-MDT0000 complete [ 4803.552124] LustreError: 218907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.556823] LustreError: 218907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4804.621861] LustreError: 218781:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738000 with bad export cookie 16386694734829423328 [ 4804.624852] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4804.625460] LustreError: 218781:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4804.750187] Lustre: server umount lustre-MDT0001 complete [ 4808.480180] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4809.527417] Key type lgssc unregistered [ 4809.672485] LNet: 220997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4809.675972] LNetError: 220997:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4809.684475] LNet: Removed LNI 192.168.204.157@tcp [ 4810.002678] Key type .llcrypt unregistered [ 4810.004491] Key type ._llcrypt unregistered [ 4819.140077] Key type ._llcrypt registered [ 4819.141927] Key type .llcrypt registered [ 4819.184977] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4819.607530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4819.624115] alg: No test for adler32 (adler32-zlib) [ 4820.483317] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4820.573609] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4822.159174] Key type lgssc registered [ 4822.522907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4825.193694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4825.201288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.364672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4827.768796] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4830.763872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4830.893066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4832.212543] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4833.961553] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4835.142796] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4837.536526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4837.671332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4839.680342] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_default_debug -1 all [ 4841.724122] Lustre: DEBUG MARKER: oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4861.409678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4861.411657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4861.412119] LustreError: Skipped 1 previous similar message [ 4861.417756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4861.420953] Lustre: Skipped 1 previous similar message [ 4863.214248] LustreError: 224787:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4863.262273] Lustre: server umount lustre-OST0000 complete [ 4866.529581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.535179] Lustre: Skipped 1 previous similar message [ 4866.537457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4866.539493] Lustre: Skipped 1 previous similar message [ 4871.648134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4871.651358] Lustre: Skipped 1 previous similar message [ 4876.768830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4876.772054] Lustre: Skipped 1 previous similar message [ 4879.327140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4879.360290] LustreError: 224992:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4879.362651] LustreError: 224992:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4879.392411] Lustre: server umount lustre-MDT0000 complete [ 4880.820714] LustreError: 222347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738076 with bad export cookie 3142664639228509973 [ 4880.824103] LustreError: MGC192.168.204.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4880.824492] LustreError: 222347:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4880.896442] LustreError: 225208:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4880.899035] LustreError: 225208:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4880.969573] Lustre: server umount lustre-MDT0001 complete [ 4884.897780] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4886.094424] Key type lgssc unregistered [ 4886.226620] LNet: 225739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4886.230499] LNetError: 225739:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4886.241476] LNet: Removed LNI 192.168.204.157@tcp [ 4886.610134] Key type .llcrypt unregistered [ 4886.611809] Key type ._llcrypt unregistered [ 4894.778477] Key type ._llcrypt registered [ 4894.779791] Key type .llcrypt registered [ 4894.830411] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4894.966352] Key type .llcrypt unregistered [ 4894.967955] Key type ._llcrypt unregistered [ 4904.819992] Key type ._llcrypt registered [ 4904.820964] Key type .llcrypt registered [ 4904.863105] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing set_hostid [ 4911.973682] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing load_modules_local [ 4912.419442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4912.427754] alg: No test for adler32 (adler32-zlib) [ 4913.314233] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4913.435508] LNet: Added LNI 192.168.204.157@tcp [8/256/0/180] [ 4915.039168] Key type lgssc registered [ 4915.458139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4918.850524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4921.894366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4924.535561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4927.142549] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4927.875185] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 4670 sec ========== 00:02:03 (1764738123) [ 4928.622820] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:02:03 (1764738123) === [ 4930.157224] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:02:05 (1764738125) === [ 4945.893774] Lustre: DEBUG MARKER: oleg457-server.virtnet: executing unload_modules_local [ 4947.112530] Key type lgssc unregistered [ 4947.258630] LNet: 231820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4947.262775] LNetError: 231820:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4947.272380] LNet: Removed LNI 192.168.204.157@tcp [ 4947.618603] Key type .llcrypt unregistered [ 4947.620184] Key type ._llcrypt unregistered