[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485015591 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002427] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007436] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.012022] Yama: becoming mindful. [ 0.012894] SELinux: Initializing. [ 0.013064] *** VALIDATE selinux *** [ 0.020787] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024662] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028115] *** VALIDATE tmpfs *** [ 0.030227] *** VALIDATE proc *** [ 0.031220] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036033] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040267] debug: unmapping init [mem 0xffffffff97859000-0xffffffff97860fff] [ 0.042164] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043684] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044023] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047016] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051320] rcu: Hierarchical SRCU implementation. [ 0.053574] smp: Bringing up secondary CPUs ... [ 0.054634] x86: Booting SMP configuration: [ 0.055023] .... node #0, CPUs: #1 #2 #3 [ 0.058644] smp: Brought up 1 node, 4 CPUs [ 0.060014] smpboot: Max logical packages: 1 [ 0.061025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148481] node 0 deferred pages initialised in 84ms [ 0.152126] devtmpfs: initialized [ 0.153226] x86/mm: Memory block size: 128MB [ 0.155795] gcov: version magic: 0x41383552 [ 0.157163] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.159306] pinctrl core: initialized pinctrl subsystem [ 0.160259] [ 0.161019] ************************************************************* [ 0.162018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163013] ** ** [ 0.164013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165019] ** ** [ 0.166066] ** This means that this kernel is built to expose internal ** [ 0.167019] ** IOMMU data structures, which may compromise security on ** [ 0.168012] ** your system. ** [ 0.169013] ** ** [ 0.170014] ** If you see this message and you are not debugging the ** [ 0.171015] ** kernel, report this immediately to your vendor! ** [ 0.172009] ** ** [ 0.173010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174012] ************************************************************* [ 0.175898] NET: Registered protocol family 16 [ 0.176480] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179504] cpuidle: using governor menu [ 0.181660] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185508] PCI: Using configuration type 1 for base access [ 0.187133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196097] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199025] cryptd: max_cpu_qlen set to 1000 [ 0.201156] ACPI: Added _OSI(Module Device) [ 0.202014] ACPI: Added _OSI(Processor Device) [ 0.204016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210223] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.217360] ACPI: Interpreter enabled [ 0.219079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221016] ACPI: Using IOAPIC for interrupt routing [ 0.222112] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227416] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237484] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.243022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250349] acpiphp: Slot [2] registered [ 0.252101] acpiphp: Slot [5] registered [ 0.254243] acpiphp: Slot [6] registered [ 0.256133] acpiphp: Slot [7] registered [ 0.257112] acpiphp: Slot [8] registered [ 0.258112] acpiphp: Slot [9] registered [ 0.260135] acpiphp: Slot [10] registered [ 0.261140] acpiphp: Slot [3] registered [ 0.263150] acpiphp: Slot [4] registered [ 0.264107] acpiphp: Slot [11] registered [ 0.266113] acpiphp: Slot [12] registered [ 0.268161] acpiphp: Slot [13] registered [ 0.269122] acpiphp: Slot [14] registered [ 0.271118] acpiphp: Slot [15] registered [ 0.272117] acpiphp: Slot [16] registered [ 0.273166] acpiphp: Slot [17] registered [ 0.274021] acpiphp: Slot [18] registered [ 0.275113] acpiphp: Slot [19] registered [ 0.277138] acpiphp: Slot [20] registered [ 0.279139] acpiphp: Slot [21] registered [ 0.280115] acpiphp: Slot [22] registered [ 0.282192] acpiphp: Slot [23] registered [ 0.284102] acpiphp: Slot [24] registered [ 0.285211] acpiphp: Slot [25] registered [ 0.287101] acpiphp: Slot [26] registered [ 0.288129] acpiphp: Slot [27] registered [ 0.290141] acpiphp: Slot [28] registered [ 0.291101] acpiphp: Slot [29] registered [ 0.293124] acpiphp: Slot [30] registered [ 0.294108] acpiphp: Slot [31] registered [ 0.296153] PCI host bridge to bus 0000:00 [ 0.297022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.299028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.302028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.305028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314249] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.317604] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.327829] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.333668] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.335028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.338024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.341020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.344587] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348006] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354847] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.359010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.370012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.376017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.383053] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.396017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.407016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.424018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438254] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447071] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.454015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.473015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.484042] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.493016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.500019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.518016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.527458] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.534017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.538020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.556017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.571273] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.586015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.597022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.618019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.628948] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.635017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.645012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.669014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.680143] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.684433] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.687344] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.689394] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.693254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.697197] iommu: Default domain type: Passthrough [ 0.700594] SCSI subsystem initialized [ 0.702149] ACPI: bus type USB registered [ 0.703117] usbcore: registered new interface driver usbfs [ 0.705069] usbcore: registered new interface driver hub [ 0.707072] usbcore: registered new device driver usb [ 0.709195] pps_core: LinuxPPS API ver. 1 registered [ 0.711012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.714080] PTP clock support registered [ 0.716103] EDAC MC: Ver: 3.0.0 [ 0.719177] PCI: Using ACPI for IRQ routing [ 0.722983] NetLabel: Initializing [ 0.724013] NetLabel: domain hash size = 128 [ 0.727016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.730113] NetLabel: unlabeled traffic allowed by default [ 0.733165] vgaarb: loaded [ 0.735348] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.737017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.744000] clocksource: Switched to clocksource kvm-clock [ 0.854346] VFS: Disk quotas dquot_6.6.0 [ 0.856046] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.859043] *** VALIDATE ramfs *** [ 0.860496] *** VALIDATE hugetlbfs *** [ 0.862339] pnp: PnP ACPI init [ 0.865084] pnp: PnP ACPI: found 6 devices [ 0.888868] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.892269] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.894176] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.896545] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.899191] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.901392] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.904448] NET: Registered protocol family 2 [ 0.907025] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.912613] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.916238] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.921138] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.924761] TCP: Hash tables configured (established 65536 bind 65536) [ 0.928209] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.931267] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.934073] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.937230] NET: Registered protocol family 1 [ 0.939756] RPC: Registered named UNIX socket transport module. [ 0.941867] RPC: Registered udp transport module. [ 0.943550] RPC: Registered tcp transport module. [ 0.947307] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.949905] NET: Registered protocol family 44 [ 0.951488] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.953448] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.955393] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.957554] PCI: CLS 0 bytes, default 64 [ 0.959600] Unpacking initramfs... [ 2.367788] debug: unmapping init [mem 0xffff8bccfcc54000-0xffff8bccfffbffff] [ 2.372269] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.375013] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.378373] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.885051] Initialise system trusted keyrings [ 2.887782] Key type blacklist registered [ 2.890825] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.901560] zbud: loaded [ 2.904673] *** VALIDATE nfs *** [ 2.906034] *** VALIDATE nfs4 *** [ 2.907827] pstore: using deflate compression [ 2.911492] Platform Keyring initialized [ 3.007890] NET: Registered protocol family 38 [ 3.009453] Key type asymmetric registered [ 3.010928] Asymmetric key parser 'x509' registered [ 3.012856] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.015712] io scheduler mq-deadline registered [ 3.017239] io scheduler kyber registered [ 3.018749] io scheduler bfq registered [ 3.020632] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.024132] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.026407] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.029296] ACPI: Power Button [PWRF] [ 3.034322] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.041435] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.057436] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.064895] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.081495] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.110080] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.138866] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.144605] Non-volatile memory driver v1.3 [ 3.146616] Linux agpgart interface v0.103 [ 3.180987] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.184621] vda: detected capacity change from 0 to 76779520 [ 3.198094] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.201864] vdb: detected capacity change from 0 to 1073741824 [ 3.215911] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.220447] vdc: detected capacity change from 0 to 2621440000 [ 3.238307] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.241026] vdd: detected capacity change from 0 to 2621440000 [ 3.256451] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.259229] vde: detected capacity change from 0 to 4294967296 [ 3.274619] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.277408] vdf: detected capacity change from 0 to 4294967296 [ 3.283531] libphy: Fixed MDIO Bus: probed [ 3.290700] usbcore: registered new interface driver usbserial_generic [ 3.293529] usbserial: USB Serial support registered for generic [ 3.296171] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.300451] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.302471] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.305047] mousedev: PS/2 mouse device common for all mice [ 3.308324] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.310692] rtc_cmos 00:05: RTC can wake from S4 [ 3.315774] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.319132] rtc_cmos 00:05: registered as rtc0 [ 3.322758] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.323764] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.325204] intel_pstate: CPU model not supported [ 3.330550] hid: raw HID events driver (C) Jiri Kosina [ 3.332721] usbcore: registered new interface driver usbhid [ 3.335043] usbhid: USB HID core driver [ 3.336968] drop_monitor: Initializing network drop monitor service [ 3.339602] Initializing XFRM netlink socket [ 3.341835] NET: Registered protocol family 10 [ 3.344225] Segment Routing with IPv6 [ 3.345563] NET: Registered protocol family 17 [ 3.347149] mpls_gso: MPLS GSO support [ 3.350296] RAS: Correctable Errors collector initialized. [ 3.351859] AVX version of gcm_enc/dec engaged. [ 3.353046] AES CTR mode by8 optimization enabled [ 3.422329] sched_clock: Marking stable (3422277021, 0)->(4391736045, -969459024) [ 3.426424] registered taskstats version 1 [ 3.428392] Loading compiled-in X.509 certificates [ 3.430589] zswap: loaded using pool lzo/zbud [ 3.457543] Key type big_key registered [ 3.470618] Key type encrypted registered [ 3.472813] ima: No TPM chip found, activating TPM-bypass! [ 3.475619] ima: Allocated hash algorithm: sha1 [ 3.477918] ima: No architecture policies found [ 3.480307] evm: Initialising EVM extended attributes: [ 3.482418] evm: security.selinux [ 3.483729] evm: security.ima [ 3.485229] evm: security.capability [ 3.486637] evm: HMAC attrs: 0x1 [ 3.489210] rtc_cmos 00:05: setting system clock to 2026-09-16 00:24:00 UTC (1789518240) [ 3.495210] debug: unmapping init [mem 0xffffffff98803000-0xffffffff989fffff] [ 3.498217] debug: unmapping init [mem 0xffffffff97582000-0xffffffff97858fff] [ 3.506168] Write protecting the kernel read-only data: 28672k [ 3.509706] debug: unmapping init [mem 0xffffffff95c03000-0xffffffff95dfffff] [ 3.512833] debug: unmapping init [mem 0xffffffff96514000-0xffffffff965fffff] [ 3.549116] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.556638] systemd[1]: Detected virtualization kvm. [ 3.558345] systemd[1]: Detected architecture x86-64. [ 3.559896] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.584253] systemd[1]: No hostname configured. [ 3.585677] systemd[1]: Set hostname to . [ 3.587472] random: systemd: uninitialized urandom read (16 bytes read) [ 3.590015] systemd[1]: Initializing machine ID from random generator. [ 3.630502] random: ln: uninitialized urandom read (6 bytes read) [ 3.711527] random: systemd: uninitialized urandom read (16 bytes read) [ 3.713777] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.717266] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.720617] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.347708] device-mapper: uevent: version 1.0.3 [ 4.350294] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.206864] random: fast init done [ 5.213330] virtio_net virtio0 ens2: renamed from eth0 [ 5.254566] scsi host0: ata_piix [ 5.268605] scsi host1: ata_piix [ 5.274790] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.278022] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.150962] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.994064] random: crng init done [ 9.996202] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.553485] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.791241] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.061841] SELinux: Disabled at runtime. [ 12.131136] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.158118] systemd[1]: Detected virtualization kvm. [ 12.160224] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.727514] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.731237] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.738584] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.742855] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.745977] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.752614] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.758648] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message [ 12.837615] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.233784] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.598664] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.708040] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.830893] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.846284] EDAC sbridge: Ver: 1.1.2 [ 15.488127] Key type dns_resolver registered [ 15.795506] NFS: Registering the id_resolver key type [ 15.797753] Key type id_resolver registered [ 15.799675] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg435-server login: [ 47.054587] libcfs: loading out-of-tree module taints kernel. [ 47.111653] Key type ._llcrypt registered [ 47.114628] Key type .llcrypt registered [ 47.238545] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 64.978815] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 66.239167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.261988] alg: No test for adler32 (adler32-zlib) [ 67.451720] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 68.203409] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 69.952975] Key type lgssc registered [ 71.404248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.397336] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 129.623183] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 145.510813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 145.615384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.194484] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 147.263499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 147.380125] Lustre: lustre-MDT0000: new disk, initializing [ 147.524544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 147.569653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.203679] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.956020] hrtimer: interrupt took 2277241 ns [ 169.666206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.757863] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.782724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.787941] Lustre: Skipped 1 previous similar message [ 169.853297] Lustre: lustre-MDT0001: new disk, initializing [ 169.905472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.930655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 169.935096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.379192] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.694364] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 190.238074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.676962] Lustre: lustre-OST0000: new disk, initializing [ 190.681614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.694382] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.751924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.232763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.246880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.357975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.301871] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.119889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.244392] Lustre: lustre-OST0001: new disk, initializing [ 213.250445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 213.255909] Lustre: 9527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.338752] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.946321] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.201222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 221.207250] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 221.282391] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.574654] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.564600] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.960212] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing check_logdir /tmp/testlogs/ [ 252.289301] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing yml_node [ 256.630098] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 259.545876] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 262.471380] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 264.699956] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 20:28:20 EDT 2026 [ 285.821832] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 287.078774] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 303.072516] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.083525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 303.083744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 303.106643] Lustre: Skipped 3 previous similar messages [ 303.139144] Lustre: Skipped 3 previous similar messages [ 308.194810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 308.210372] Lustre: Skipped 3 previous similar messages [ 308.587661] Lustre: server umount lustre-MDT0000 complete [ 312.030624] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518549 with bad export cookie 17734567012694359859 [ 312.041873] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.313118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.316912] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.328598] Lustre: Skipped 1 previous similar message [ 317.354729] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.363407] Lustre: Skipped 1 previous similar message [ 318.448384] Lustre: server umount lustre-MDT0001 complete [ 322.959655] Lustre: server umount lustre-OST0000 complete [ 327.919457] Lustre: server umount lustre-OST0001 complete [ 334.231525] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 342.555788] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 376.199976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.434581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 376.473967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 376.583519] Lustre: lustre-MDT0000: new disk, initializing [ 376.663748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 376.686449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 381.324864] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 391.519314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.666498] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 391.738816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 391.743396] Lustre: Skipped 1 previous similar message [ 391.840760] Lustre: lustre-MDT0001: new disk, initializing [ 391.966675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 392.005227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 392.012925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.911927] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 405.196087] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 410.911939] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 417.859238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.132515] Lustre: lustre-OST0000: new disk, initializing [ 418.136664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 418.147258] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 418.276100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.400574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 419.411906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 419.533869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 424.349474] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 433.826160] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 437.825521] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 438.042396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 442.355204] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 442.591904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 445.408865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 445.414861] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.425748] LustreError: Skipped 1 previous similar message [ 445.426627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 445.474771] Lustre: Skipped 1 previous similar message [ 450.683679] Lustre: server umount lustre-OST0000 complete [ 455.659694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.663853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 455.685437] Lustre: Skipped 1 previous similar message [ 455.707658] Lustre: Skipped 4 previous similar messages [ 461.818487] Lustre: server umount lustre-MDT0000 complete [ 465.664120] LustreError: 17046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518702 with bad export cookie 17734567012694361392 [ 465.671028] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.680016] LustreError: 17046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 466.125595] Lustre: server umount lustre-MDT0001 complete [ 469.451755] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 20:31:45 (1789518705) [ 477.212600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 477.708423] LustreError: 19314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.735267] LustreError: 19314:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 477.818346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 482.608595] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 483.296893] LustreError: 19315:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.398510] LustreError: 19314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.929625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.165556] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 502.032844] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 507.819662] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 513.965195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 519.681175] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 526.846966] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 537.061348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 537.070313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.085083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.088317] Lustre: Skipped 2 previous similar messages [ 539.651372] Lustre: server umount lustre-OST0000 complete [ 549.298150] Lustre: server umount lustre-MDT0000 complete [ 552.802173] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518789 with bad export cookie 17734567012694362274 [ 552.806388] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.828213] LustreError: 19296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 553.182800] Lustre: server umount lustre-MDT0001 complete [ 564.996735] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 568.667765] Key type lgssc unregistered [ 569.034246] LNet: 22845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.045738] LNetError: 22845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 569.073208] LNet: Removed LNI 192.168.204.135@tcp [ 570.063105] Key type .llcrypt unregistered [ 570.068082] Key type ._llcrypt unregistered [ 594.704458] Key type ._llcrypt registered [ 594.707389] Key type .llcrypt registered [ 594.821064] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 596.228397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 596.392285] alg: No test for adler32 (adler32-zlib) [ 597.511534] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 597.792692] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 599.463758] Key type lgssc registered [ 600.648940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 608.671085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 608.689599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.271710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.769574] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 620.512558] LustreError: 24253:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.530044] LustreError: 24253:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 625.642789] LustreError: 24254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.740485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.412104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 632.739819] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 642.662832] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 648.448298] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 656.308815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 656.882441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.308507] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 673.524295] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 680.495666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 690.663955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 690.667304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.710504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 690.718075] Lustre: Skipped 1 previous similar message [ 695.782345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 696.474885] Lustre: server umount lustre-OST0000 complete [ 703.460412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 703.472864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.498802] Lustre: Skipped 1 previous similar message [ 703.523897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 703.536520] Lustre: Skipped 1 previous similar message [ 706.016876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.020737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.264310] Lustre: server umount lustre-MDT0000 complete [ 711.139539] LustreError: 24254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.170956] LustreError: 24254:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 711.193297] LustreError: 24235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789518948 with bad export cookie 17359826054951684869 [ 711.205413] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 711.828954] Lustre: server umount lustre-MDT0001 complete [ 723.763630] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 726.675468] Key type lgssc unregistered [ 726.924690] LNet: 28231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.937564] LNetError: 28231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.958394] LNet: Removed LNI 192.168.204.135@tcp [ 728.016906] Key type .llcrypt unregistered [ 728.024218] Key type ._llcrypt unregistered [ 754.636043] Key type ._llcrypt registered [ 754.638830] Key type .llcrypt registered [ 754.797834] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 756.005392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 756.145759] alg: No test for adler32 (adler32-zlib) [ 757.289842] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 757.586804] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 759.335181] Key type lgssc registered [ 760.946126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 769.276991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 769.306414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 770.835601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.650246] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 780.260518] LustreError: 29647:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 780.278879] LustreError: 29647:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 784.973620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.334698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 789.842585] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 796.686684] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 801.797910] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 814.310724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 814.567854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.597071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 819.910085] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 827.783854] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 841.197157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 841.208018] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.211854] LustreError: Skipped 1 previous similar message [ 841.219980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 841.228014] Lustre: Skipped 1 previous similar message [ 841.274583] Lustre: Skipped 1 previous similar message [ 846.313434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 846.325480] Lustre: Skipped 1 previous similar message [ 847.467342] Lustre: server umount lustre-OST0000 complete [ 856.546351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.582058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 856.587469] Lustre: Skipped 1 previous similar message [ 859.908403] Lustre: server umount lustre-MDT0000 complete [ 861.666731] LustreError: 30660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 861.694437] LustreError: 30660:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 863.826830] LustreError: 29626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519100 with bad export cookie 4853606792158260365 [ 863.847237] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 863.855400] LustreError: 29626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 864.270156] Lustre: server umount lustre-MDT0001 complete [ 875.021438] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 878.161332] Key type lgssc unregistered [ 878.354607] LNet: 33544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.361057] LNetError: 33544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.400333] LNet: Removed LNI 192.168.204.135@tcp [ 880.189103] Key type .llcrypt unregistered [ 880.194256] Key type ._llcrypt unregistered [ 904.771734] Key type ._llcrypt registered [ 904.775512] Key type .llcrypt registered [ 904.901613] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 906.201110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 906.294820] alg: No test for adler32 (adler32-zlib) [ 907.500861] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 907.799625] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 909.479221] Key type lgssc registered [ 910.696780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 919.873779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 919.929926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 921.301089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 926.163794] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 931.810097] LustreError: 34958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.833845] LustreError: 34958:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 935.147039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.642850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 939.848380] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 947.960632] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.437327] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 960.078410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 960.441569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.388428] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 966.659530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 974.583423] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 986.080367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 986.087058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.103119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 987.114569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 987.118017] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.152292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 992.240090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 992.246613] Lustre: Skipped 2 previous similar messages [ 997.354141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 997.364406] Lustre: Skipped 1 previous similar message [ 998.369070] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 998.759275] Lustre: server umount lustre-OST0000 complete [ 1007.587325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.595985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1007.605930] Lustre: Skipped 1 previous similar message [ 1007.606469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1007.628138] Lustre: Skipped 1 previous similar message [ 1017.824512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.833156] Lustre: Skipped 3 previous similar messages [ 1018.335634] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1018.529674] Lustre: server umount lustre-MDT0000 complete [ 1022.945692] LustreError: 37450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1022.970315] LustreError: 37450:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1023.744511] LustreError: 34938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519260 with bad export cookie 18263331001258141930 [ 1023.762620] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.766122] LustreError: 34938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1024.312897] Lustre: server umount lustre-MDT0001 complete [ 1038.086261] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1041.149711] Key type lgssc unregistered [ 1041.495578] LNet: 38496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.512646] LNetError: 38496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.528363] LNet: Removed LNI 192.168.204.135@tcp [ 1042.410672] Key type .llcrypt unregistered [ 1042.412943] Key type ._llcrypt unregistered [ 1068.282163] Key type ._llcrypt registered [ 1068.292871] Key type .llcrypt registered [ 1068.510801] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1069.942221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1070.046047] alg: No test for adler32 (adler32-zlib) [ 1071.172399] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1071.545340] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1073.303426] Key type lgssc registered [ 1074.710262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1083.122505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1083.139341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.610418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.683526] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1094.114169] LustreError: 39908:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.150286] LustreError: 39908:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1098.983796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.243094] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1099.454599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1104.047072] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1112.452036] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1118.459219] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1127.723566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1128.213859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.263114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1134.984791] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1143.281367] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1149.924188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1149.928401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.949449] LustreError: Skipped 1 previous similar message [ 1149.954736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1149.962297] Lustre: Skipped 1 previous similar message [ 1149.981645] Lustre: Skipped 1 previous similar message [ 1155.045520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1155.051019] Lustre: Skipped 1 previous similar message [ 1155.979957] Lustre: server umount lustre-OST0000 complete [ 1180.641639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.661746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.666804] Lustre: Skipped 3 previous similar messages [ 1186.785299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1186.799034] Lustre: Skipped 3 previous similar messages [ 1191.392035] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1191.580486] Lustre: server umount lustre-MDT0000 complete [ 1191.905897] LustreError: 40922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1191.931676] LustreError: 40922:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1195.594951] LustreError: 39889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519432 with bad export cookie 5318443278101134002 [ 1195.595843] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1195.605825] LustreError: 39889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1202.081894] Lustre: server umount lustre-MDT0001 complete [ 1217.229392] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1220.956492] Key type lgssc unregistered [ 1221.311947] LNet: 43619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.321310] LNetError: 43619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.335585] LNet: Removed LNI 192.168.204.135@tcp [ 1222.359163] Key type .llcrypt unregistered [ 1222.362679] Key type ._llcrypt unregistered [ 1247.155444] Key type ._llcrypt registered [ 1247.159905] Key type .llcrypt registered [ 1247.307867] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1248.452265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1248.763909] alg: No test for adler32 (adler32-zlib) [ 1249.938199] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1250.246908] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1251.960297] Key type lgssc registered [ 1253.242245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1261.295695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1261.321280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.803276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.289787] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1272.299710] LustreError: 45035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1272.323300] LustreError: 45035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1276.192677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.458500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1281.353113] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1289.466509] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1296.798712] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1304.725355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1305.315543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.424463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1313.826581] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1323.330699] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1331.086917] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [ 1332.199836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.209110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1332.215515] Lustre: Skipped 1 previous similar message [ 1336.153287] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.35@tcp (stopping) [ 1336.179267] Lustre: Skipped 2 previous similar messages [ 1336.443713] Lustre: server umount lustre-MDT0000 complete [ 1337.321054] LustreError: 45031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1337.336336] LustreError: 45031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1340.519965] LustreError: 45016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789519577 with bad export cookie 5394757451316108509 [ 1340.525873] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.540248] LustreError: 45016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1341.278632] LustreError: 45030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.315133] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.35@tcp (stopping) [ 1342.450054] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.462157] Lustre: Skipped 1 previous similar message [ 1345.442553] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1345.442739] LustreError: 45030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.448619] Lustre: Skipped 1 previous similar message [ 1345.471659] LustreError: 45030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1347.129979] Lustre: server umount lustre-MDT0001 complete [ 1368.662258] Lustre: server umount lustre-OST0000 complete [ 1383.159968] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1385.847230] Key type lgssc unregistered [ 1386.143369] LNet: 48893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1386.150355] LNetError: 48893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1386.166542] LNet: Removed LNI 192.168.204.135@tcp [ 1386.978395] Key type .llcrypt unregistered [ 1386.980226] Key type ._llcrypt unregistered [ 1409.705121] Key type ._llcrypt registered [ 1409.714631] Key type .llcrypt registered [ 1409.844854] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1410.846314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.976590] alg: No test for adler32 (adler32-zlib) [ 1412.060238] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1412.320295] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1413.975283] Key type lgssc registered [ 1414.808897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1421.603079] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1421.621512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1437.152298] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1442.272202] LustreError: 50275:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 1468.959549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.191580] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1486.079157] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1653.662433] Lustre: server umount lustre-OST0000 complete [ 1670.771774] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1673.739685] Key type lgssc unregistered [ 1674.105402] LNet: 51951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.122225] LNetError: 51951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.146987] LNet: Removed LNI 192.168.204.135@tcp [ 1675.281632] Key type .llcrypt unregistered [ 1675.284110] Key type ._llcrypt unregistered [ 1702.409259] Key type ._llcrypt registered [ 1702.412525] Key type .llcrypt registered [ 1702.501453] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1703.728890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1703.770431] alg: No test for adler32 (adler32-zlib) [ 1705.143392] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1705.433317] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1707.280080] Key type lgssc registered [ 1708.495925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1718.022924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1718.067687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.734583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.319207] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1728.999928] LustreError: 53365:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1729.016804] LustreError: 53365:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1734.117266] LustreError: 53364:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.876416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.416547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1743.105285] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1751.789833] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1758.093219] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1765.687703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1766.079470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1772.692434] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1781.182969] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1788.901891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1788.908556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.922211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1789.997083] LustreError: 55185:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1790.154092] Lustre: server umount lustre-OST0000 complete [ 1794.531162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1794.539112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1794.551476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.160236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.170446] Lustre: Skipped 3 previous similar messages [ 1800.572482] Lustre: server umount lustre-MDT0000 complete [ 1804.299606] LustreError: 53344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520041 with bad export cookie 445961417744960035 [ 1804.302545] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1804.306157] LustreError: 53344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1804.579254] Lustre: server umount lustre-MDT0001 complete [ 1817.308128] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 1820.134081] Key type lgssc unregistered [ 1820.409923] LNet: 56895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.417191] LNetError: 56895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.439223] LNet: Removed LNI 192.168.204.135@tcp [ 1821.509380] Key type .llcrypt unregistered [ 1821.513454] Key type ._llcrypt unregistered [ 1846.714050] Key type ._llcrypt registered [ 1846.717391] Key type .llcrypt registered [ 1846.869366] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 1848.106336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.177513] alg: No test for adler32 (adler32-zlib) [ 1849.165557] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1849.381293] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 1851.095575] Key type lgssc registered [ 1852.139314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1860.348719] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1860.381943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1875.935418] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1881.055335] LustreError: 58276:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 1907.743338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.435462] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1922.058795] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1929.642375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.123390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.246219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1935.338871] LustreError: 58301:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.368671] LustreError: 58301:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1936.404665] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1940.458466] LustreError: 58302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.503036] LustreError: 59786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.518689] LustreError: 59786:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1947.617763] LustreError: 58302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.631548] LustreError: 58302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1948.402449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.895776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1954.260735] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 1962.688081] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1968.090691] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1971.169454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1971.174333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.194942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1974.754172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1974.755971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1974.770926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.302469] Lustre: server umount lustre-OST0000 complete [ 1999.332341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1999.351505] Lustre: Skipped 1 previous similar message [ 1999.362539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1999.371563] Lustre: Skipped 2 previous similar messages [ 2004.448752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2004.797696] Lustre: server umount lustre-MDT0000 complete [ 2008.045505] LustreError: 58284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520245 with bad export cookie 5658482316957140916 [ 2008.048799] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2008.062257] LustreError: 58284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2022.369600] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2022.664796] Lustre: server umount lustre-MDT0001 complete [ 2034.602916] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2037.464543] Key type lgssc unregistered [ 2037.769478] LNet: 62023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.777622] LNetError: 62023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.823738] LNet: Removed LNI 192.168.204.135@tcp [ 2039.726719] Key type .llcrypt unregistered [ 2039.729719] Key type ._llcrypt unregistered [ 2064.538509] Key type ._llcrypt registered [ 2064.543502] Key type .llcrypt registered [ 2064.692673] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2065.512255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2065.536234] alg: No test for adler32 (adler32-zlib) [ 2066.708624] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2066.994342] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2068.719248] Key type lgssc registered [ 2070.061652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2078.563864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2078.605395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.281163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.937951] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2090.466870] LustreError: 63433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.496399] LustreError: 63433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2094.492715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.857572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2099.865332] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2108.204287] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2114.316476] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2122.741164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2123.170944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.240814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2129.566800] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2137.682812] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2170.339624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2170.362487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.393202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2170.398897] Lustre: Skipped 1 previous similar message [ 2170.416888] Lustre: Skipped 1 previous similar message [ 2173.350976] Lustre: server umount lustre-OST0000 complete [ 2180.589475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2180.595178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2180.612552] Lustre: Skipped 1 previous similar message [ 2185.704551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.713086] Lustre: Skipped 2 previous similar messages [ 2190.826983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.837361] Lustre: Skipped 1 previous similar message [ 2193.375131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2193.584871] Lustre: server umount lustre-MDT0000 complete [ 2195.941486] LustreError: 63430:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.962789] LustreError: 63430:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2197.399548] LustreError: 63413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520434 with bad export cookie 7945570769701028189 [ 2197.403628] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.408552] LustreError: 63413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2197.829958] Lustre: server umount lustre-MDT0001 complete [ 2210.380481] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2213.677390] Key type lgssc unregistered [ 2214.068670] LNet: 66993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.077808] LNetError: 66993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.152591] LNet: Removed LNI 192.168.204.135@tcp [ 2216.224473] Key type .llcrypt unregistered [ 2216.229591] Key type ._llcrypt unregistered [ 2242.833705] Key type ._llcrypt registered [ 2242.837523] Key type .llcrypt registered [ 2242.958048] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2244.146680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2244.187570] alg: No test for adler32 (adler32-zlib) [ 2245.270765] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2245.503396] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2247.207641] Key type lgssc registered [ 2248.507200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.381469] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 21:01:33 (1789520493) [ 2264.790429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2264.811378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.314098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.199277] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2276.859962] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.895096] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2280.455853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.853816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2285.717330] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2294.415400] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2301.065926] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2309.175284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2309.615878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.691668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2315.168233] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2324.763857] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2331.150060] Lustre: Failing over lustre-MDT0000 [ 2331.415245] Lustre: server umount lustre-MDT0000 complete [ 2331.569464] LustreError: 68701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.136758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2332.150994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.180954] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.189101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.210178] LustreError: 68705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2336.607987] LustreError: 68702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.648244] LustreError: 68702:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2341.746792] LustreError: 68700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.775114] LustreError: 68700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2342.166107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.404815] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2342.674461] Lustre: *** cfs_fail_loc=135, val=0*** [ 2342.789095] LustreError: 71794:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2342.794449] LustreError: 71794:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -2 [ 2342.805884] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2342.805884] [ 2342.815815] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2342.825291] LustreError: 71782:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2342.830603] LustreError: 71782:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2342.842538] LustreError: 71782:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2342.850052] LustreError: 71782:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2342.873469] Lustre: server umount lustre-MDT0000 complete [ 2342.880600] LustreError: 71782:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2350.011736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.101955] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2350.117579] LustreError: 68697:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2350.386069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.447072] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2351.963699] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2355.247547] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2355.738309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2355.775606] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2355.844225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2365.924687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2365.937480] Lustre: Skipped 1 previous similar message [ 2365.944401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2365.952671] Lustre: Skipped 1 previous similar message [ 2369.335399] Lustre: server umount lustre-OST0000 complete [ 2376.160844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2376.166167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.172199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2376.181549] Lustre: Skipped 2 previous similar messages [ 2376.195718] Lustre: Skipped 1 previous similar message [ 2379.403922] Lustre: server umount lustre-MDT0000 complete [ 2381.280444] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.320914] LustreError: 68706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 2383.238902] LustreError: 68686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520620 with bad export cookie 7174454935116393551 [ 2383.245177] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.248385] LustreError: 68686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2383.642736] Lustre: server umount lustre-MDT0001 complete [ 2394.453553] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2397.016465] Key type lgssc unregistered [ 2397.268059] LNet: 73573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.273518] LNetError: 73573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.286662] LNet: Removed LNI 192.168.204.135@tcp [ 2398.149806] Key type .llcrypt unregistered [ 2398.151845] Key type ._llcrypt unregistered [ 2420.561121] Key type ._llcrypt registered [ 2420.565525] Key type .llcrypt registered [ 2420.678473] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2422.092061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2422.223693] alg: No test for adler32 (adler32-zlib) [ 2423.341809] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2423.611102] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2425.343376] Key type lgssc registered [ 2426.679217] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2435.949218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2435.993122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.564283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.510323] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2447.842775] LustreError: 74981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.862719] LustreError: 74981:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2452.217038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.848559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.848683] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2466.251615] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2473.833759] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2481.941601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2482.405449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.450833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2490.284207] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2498.598163] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2505.152075] Lustre: Failing over lustre-MDT0000 [ 2505.525145] Lustre: server umount lustre-MDT0000 complete [ 2506.619803] LustreError: 74976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.260362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.264897] LustreError: 74977:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.276898] Lustre: Skipped 2 previous similar messages [ 2508.294210] LustreError: 74977:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2511.705796] LustreError: 74978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.755091] LustreError: 74978:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2515.181485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.333936] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2515.489336] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2515.627412] LustreError: 78069:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2515.637646] LustreError: 78069:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -9 [ 2515.651885] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2515.651885] [ 2515.664403] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2515.698767] LustreError: 78057:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2515.712637] LustreError: 78057:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2515.726958] LustreError: 78057:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2515.737171] LustreError: 78057:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2515.786690] Lustre: server umount lustre-MDT0000 complete [ 2515.790144] LustreError: 78057:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2516.842546] LustreError: 74976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.864690] LustreError: 74976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2523.776620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.946485] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.957515] LustreError: 74973:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2524.241468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.291340] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2527.075351] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2529.280352] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2529.284519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2529.338559] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2529.415131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2538.983925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2539.003585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2539.040300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2542.520222] Lustre: server umount lustre-OST0000 complete [ 2549.730389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.764559] Lustre: Skipped 1 previous similar message [ 2549.796390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2549.808388] Lustre: Skipped 3 previous similar messages [ 2554.753590] Lustre: server umount lustre-MDT0000 complete [ 2554.858330] LustreError: 74982:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.890873] LustreError: 74982:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2559.732372] LustreError: 74961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520796 with bad export cookie 14009915919625624759 [ 2559.765125] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2559.771138] LustreError: 74961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2560.099021] Lustre: server umount lustre-MDT0001 complete [ 2572.825425] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2575.652202] Key type lgssc unregistered [ 2576.067508] LNet: 79851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.086242] LNetError: 79851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2576.109539] LNet: Removed LNI 192.168.204.135@tcp [ 2577.123089] Key type .llcrypt unregistered [ 2577.126382] Key type ._llcrypt unregistered [ 2593.082575] Key type ._llcrypt registered [ 2593.087103] Key type .llcrypt registered [ 2593.246362] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2593.564304] Key type .llcrypt unregistered [ 2593.567824] Key type ._llcrypt unregistered [ 2620.602045] Key type ._llcrypt registered [ 2620.608255] Key type .llcrypt registered [ 2620.724319] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2621.730658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2621.750200] alg: No test for adler32 (adler32-zlib) [ 2622.748621] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2623.075122] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2624.831224] Key type lgssc registered [ 2626.042377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2633.753899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2633.800489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.234208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2639.952616] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2645.473376] LustreError: 81747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.486904] LustreError: 81747:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2648.915211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.280549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2653.300146] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2661.915964] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2668.130894] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2676.663441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2676.993343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2680.165824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2684.543110] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2693.031936] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2704.865539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2704.869318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.885845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2704.915617] Lustre: Skipped 1 previous similar message [ 2709.082707] Lustre: server umount lustre-OST0000 complete [ 2715.104416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.117733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2715.121883] Lustre: Skipped 2 previous similar messages [ 2718.832760] Lustre: server umount lustre-MDT0000 complete [ 2720.229620] LustreError: 83724:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.243979] LustreError: 83724:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2722.758285] LustreError: 81728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789520959 with bad export cookie 2428896024149542461 [ 2722.765290] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2722.772055] LustreError: 81728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2723.094941] Lustre: server umount lustre-MDT0001 complete [ 2733.768140] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2736.575415] Key type lgssc unregistered [ 2736.849565] LNet: 85279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2736.862330] LNetError: 85279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2736.874213] LNet: Removed LNI 192.168.204.135@tcp [ 2737.734406] Key type .llcrypt unregistered [ 2737.737849] Key type ._llcrypt unregistered [ 2757.809136] Key type ._llcrypt registered [ 2757.811356] Key type .llcrypt registered [ 2757.895836] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2758.792628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2758.969053] alg: No test for adler32 (adler32-zlib) [ 2760.069468] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2760.374588] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2762.087265] Key type lgssc registered [ 2763.117309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2771.136659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2771.188218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2772.708228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.714781] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2782.177491] LustreError: 86669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.195665] LustreError: 86669:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2787.276717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2787.296662] LustreError: 86668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2787.857448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2793.446478] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2802.338542] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2808.919897] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2816.751996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2817.278316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.290868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2825.349567] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2833.932527] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2843.107888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.116716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2843.140532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2843.158033] Lustre: Skipped 1 previous similar message [ 2847.139347] Lustre: server umount lustre-OST0000 complete [ 2853.350527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.362948] Lustre: Skipped 2 previous similar messages [ 2853.364415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2853.375965] Lustre: Skipped 1 previous similar message [ 2857.099455] Lustre: server umount lustre-MDT0000 complete [ 2858.468746] LustreError: 86669:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2858.511698] LustreError: 86669:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2862.027514] LustreError: 87402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521099 with bad export cookie 17291546892223551483 [ 2862.043443] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2862.394570] Lustre: server umount lustre-MDT0001 complete [ 2874.203239] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 2877.544558] Key type lgssc unregistered [ 2877.925261] LNet: 90201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2877.936794] LNetError: 90201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2878.954326] LNet: Removed LNI 192.168.204.135@tcp [ 2880.068604] Key type .llcrypt unregistered [ 2880.074536] Key type ._llcrypt unregistered [ 2903.127832] Key type ._llcrypt registered [ 2903.130404] Key type .llcrypt registered [ 2903.242874] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 2904.169156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2904.304319] alg: No test for adler32 (adler32-zlib) [ 2905.311605] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2905.554035] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 2907.207189] Key type lgssc registered [ 2908.144544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.396571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2915.429771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.911690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2920.602982] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2926.055178] LustreError: 91610:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.075391] LustreError: 91610:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2928.132853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.545476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2932.413901] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2938.895304] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2944.687655] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2950.848957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2951.112806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2957.167809] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 2961.413944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2964.262303] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2975.716474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2975.727970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.744284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2976.492787] Lustre: server umount lustre-OST0000 complete [ 2981.867279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.870449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2981.875903] Lustre: Skipped 1 previous similar message [ 2981.901517] Lustre: Skipped 1 previous similar message [ 2986.449531] Lustre: server umount lustre-MDT0000 complete [ 2986.981170] LustreError: 91611:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.005148] LustreError: 91611:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2990.561632] LustreError: 91590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521227 with bad export cookie 14130198156027353990 [ 2990.570304] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2990.576538] LustreError: 91590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3000.952803] Lustre: server umount lustre-MDT0001 complete [ 3012.764141] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3015.768157] Key type lgssc unregistered [ 3016.098861] LNet: 95144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.104507] LNetError: 95144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.122456] LNet: Removed LNI 192.168.204.135@tcp [ 3016.975601] Key type .llcrypt unregistered [ 3016.979176] Key type ._llcrypt unregistered [ 3042.240825] Key type ._llcrypt registered [ 3042.244788] Key type .llcrypt registered [ 3042.428586] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3043.399302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3043.646894] alg: No test for adler32 (adler32-zlib) [ 3044.652349] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 3044.942128] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3046.688119] Key type lgssc registered [ 3047.746615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3055.692335] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3055.724806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3071.391947] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3076.511731] LustreError: 96538:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 3103.208437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.678878] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3121.364610] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3128.183834] Lustre: server umount lustre-OST0000 complete [ 3136.800297] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 21:16:12 (1789521372) [ 3144.300156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3144.963293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.116935] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3150.309990] LustreError: 98178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.342554] LustreError: 98178:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3154.401887] LustreError: 98179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.509701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.527374] LustreError: 98178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.997445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3164.657636] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3174.431059] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3181.363410] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3188.922991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.322125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.339094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3197.026889] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3204.771155] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3219.943440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3219.947122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.956125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3219.974994] Lustre: Skipped 1 previous similar message [ 3220.005650] Lustre: Skipped 1 previous similar message [ 3222.315685] Lustre: server umount lustre-OST0000 complete [ 3226.595858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3226.614768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.639068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.177582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.180341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3230.199254] Lustre: Skipped 1 previous similar message [ 3232.339584] Lustre: server umount lustre-MDT0000 complete [ 3235.297381] LustreError: 99825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.327905] LustreError: 99825:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3235.916326] LustreError: 98159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521472 with bad export cookie 9614411257792791873 [ 3235.920348] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3235.926099] LustreError: 98159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3236.237720] Lustre: server umount lustre-MDT0001 complete [ 3245.486258] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3247.577881] Key type lgssc unregistered [ 3247.838787] LNet: 101868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3247.845259] LNetError: 101868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3247.863597] LNet: Removed LNI 192.168.204.135@tcp [ 3248.544142] Key type .llcrypt unregistered [ 3248.547991] Key type ._llcrypt unregistered [ 3286.356154] Key type ._llcrypt registered [ 3286.359180] Key type .llcrypt registered [ 3286.475943] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3287.489446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3287.634253] alg: No test for adler32 (adler32-zlib) [ 3288.641158] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 3288.903969] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3290.583228] Key type lgssc registered [ 3292.228560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3302.203415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3302.220393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3303.771470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3309.142709] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3313.121558] LustreError: 104051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3313.161872] LustreError: 104051:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3318.245484] LustreError: 104052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.832137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.197648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3325.227204] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3333.628886] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3340.039379] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3348.152812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3348.698104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3349.751583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3355.637341] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3363.923133] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3375.584867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3375.600173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3375.633117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3379.626882] Lustre: server umount lustre-OST0000 complete [ 3385.830678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.854151] Lustre: Skipped 1 previous similar message [ 3385.866183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3385.875067] Lustre: Skipped 2 previous similar messages [ 3389.974881] Lustre: server umount lustre-MDT0000 complete [ 3390.948233] LustreError: 104052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.969031] LustreError: 104052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3394.701315] LustreError: 104033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521631 with bad export cookie 16066011691075303718 [ 3394.730332] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3395.266945] Lustre: server umount lustre-MDT0001 complete [ 3407.672857] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3411.927756] Key type lgssc unregistered [ 3412.364683] LNet: 107584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3412.386738] LNetError: 107584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3412.446652] LNet: Removed LNI 192.168.204.135@tcp [ 3414.396253] Key type .llcrypt unregistered [ 3414.402301] Key type ._llcrypt unregistered [ 3437.887626] Key type ._llcrypt registered [ 3437.892174] Key type .llcrypt registered [ 3438.054362] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3439.421949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3439.495940] alg: No test for adler32 (adler32-zlib) [ 3440.639957] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 3440.927143] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 3442.663287] Key type lgssc registered [ 3444.061880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3452.241408] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3452.269517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3467.808367] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3472.927338] LustreError: 108827:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 3499.616547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3505.789363] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3514.067220] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3521.335086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.625961] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3521.646420] LustreError: 109850:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3521.655741] LustreError: 109850:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3521.661980] LustreError: 109850:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3521.837781] Lustre: server umount lustre-MDT0000 complete [ 3521.854981] LustreError: 109850:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3543.007731] Lustre: 108162:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789521763/real 1789521763] req@0000000006e545ac x1876449481985280/t0(0) o400->MGC192.168.204.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789521779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3543.040763] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3543.056919] LustreError: 108839:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3563.612196] Lustre: server umount lustre-OST0000 complete [ 3573.931326] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 3583.969228] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3621.379719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.682285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3621.715987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3621.800474] Lustre: lustre-MDT0000: new disk, initializing [ 3621.891769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3621.907556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3626.841926] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3637.448478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3637.611029] Lustre: 112647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3637.683951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3637.688251] Lustre: Skipped 1 previous similar message [ 3637.796373] Lustre: lustre-MDT0001: new disk, initializing [ 3637.904897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3637.937870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3637.947866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3642.324398] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3651.181356] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3656.361884] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3662.616416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3663.042958] Lustre: lustre-OST0000: new disk, initializing [ 3663.050753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3663.060060] Lustre: 114614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3663.153342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3670.165832] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3671.092946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3671.108776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3671.248835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3678.887891] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3683.463497] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3683.693124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3688.033069] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3688.294972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3692.008329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3692.009168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.019181] LustreError: Skipped 1 previous similar message [ 3692.022471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3692.051240] Lustre: Skipped 1 previous similar message [ 3696.661258] Lustre: server umount lustre-OST0000 complete [ 3702.242730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3702.244982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3702.271107] Lustre: Skipped 1 previous similar message [ 3702.273077] Lustre: Skipped 1 previous similar message [ 3706.865922] Lustre: server umount lustre-MDT0000 complete [ 3711.224894] LustreError: 113441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789521948 with bad export cookie 1131011728511371048 [ 3711.230273] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3711.244255] LustreError: 113441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3711.690601] Lustre: server umount lustre-MDT0001 complete [ 3721.060354] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 21:25:56 (1789521956) [ 3724.794429] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3741.968566] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 3750.889676] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 3785.611414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3785.841497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3785.873653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3785.947275] Lustre: lustre-MDT0000: new disk, initializing [ 3786.040183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3786.087353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3790.767744] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3801.438656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.563706] Lustre: 119522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3801.615303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3801.623218] Lustre: Skipped 1 previous similar message [ 3801.718994] Lustre: lustre-MDT0001: new disk, initializing [ 3801.831800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3801.865067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3801.881383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3805.626761] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3813.989903] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3819.154592] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3825.917098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3826.205490] Lustre: lustre-OST0000: new disk, initializing [ 3826.208487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3826.218841] Lustre: 121490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3826.315942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3831.626764] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3834.425074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3834.437122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3834.568868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3840.931021] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3846.589247] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3847.117394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3852.359466] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3852.629271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3855.331403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3855.335467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.348311] LustreError: Skipped 1 previous similar message [ 3855.349803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.349809] Lustre: Skipped 1 previous similar message [ 3855.393043] Lustre: Skipped 1 previous similar message [ 3860.450523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3860.458086] Lustre: Skipped 2 previous similar messages [ 3861.143438] Lustre: server umount lustre-OST0000 complete [ 3868.641727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3868.654171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3868.679527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3874.222186] Lustre: server umount lustre-MDT0000 complete [ 3875.809763] LustreError: 119529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3875.827604] LustreError: 119529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3878.396918] LustreError: 119514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522115 with bad export cookie 1131011728511371930 [ 3878.405594] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3878.412885] LustreError: 119514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3879.184559] Lustre: server umount lustre-MDT0001 complete [ 3888.989875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3889.636285] LustreError: 123706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3889.723524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3894.762713] LustreError: 123707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3894.803954] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3899.883444] LustreError: 123706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.660877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3908.396883] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3915.841843] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3921.621783] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3927.998143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3934.505975] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 3943.086788] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3950.073621] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3955.194255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3955.195070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3955.197591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3955.197601] Lustre: Skipped 2 previous similar messages [ 3955.207198] Lustre: Skipped 2 previous similar messages [ 3960.570845] Lustre: server umount lustre-OST0000 complete [ 3965.408858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3965.420973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3965.457427] Lustre: Skipped 1 previous similar message [ 3971.139829] Lustre: server umount lustre-MDT0000 complete [ 3974.875305] LustreError: 123687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522211 with bad export cookie 1131011728511372812 [ 3974.879215] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3974.887500] LustreError: 123687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3975.355932] Lustre: server umount lustre-MDT0001 complete [ 3986.639824] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 3989.495814] Key type lgssc unregistered [ 3989.820971] LNet: 127335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3989.839661] LNetError: 127335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3989.879224] LNet: Removed LNI 192.168.204.135@tcp [ 3990.768072] Key type .llcrypt unregistered [ 3990.772665] Key type ._llcrypt unregistered [ 4010.818423] Key type ._llcrypt registered [ 4010.821796] Key type .llcrypt registered [ 4010.961941] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 4030.222877] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 4031.263884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4031.281350] alg: No test for adler32 (adler32-zlib) [ 4032.306704] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 4032.556637] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 4034.247994] Key type lgssc registered [ 4035.422792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4076.638888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4076.671506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4078.117940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4078.176830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4078.302341] Lustre: lustre-MDT0000: new disk, initializing [ 4078.424430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.455116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4086.044203] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4098.562186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.698925] Lustre: 130491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4098.749812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4098.754578] Lustre: Skipped 1 previous similar message [ 4098.808533] Lustre: lustre-MDT0001: new disk, initializing [ 4098.887423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4098.920830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4098.932164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4103.427367] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4112.855698] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4118.583169] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4126.340213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.702641] Lustre: lustre-OST0000: new disk, initializing [ 4126.706484] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4126.714301] Lustre: 132459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4126.808522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4133.290395] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4134.012478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4134.034803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4134.196895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4143.889271] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4150.958146] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4151.327202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4157.139940] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4157.547506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4165.088174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4165.094039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4165.096173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4165.111943] Lustre: Skipped 1 previous similar message [ 4170.215345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4170.225916] Lustre: Skipped 2 previous similar messages [ 4174.303159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4174.765664] Lustre: server umount lustre-OST0000 complete [ 4180.448816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4180.470686] Lustre: Skipped 1 previous similar message [ 4180.471597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.569563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.574898] Lustre: Skipped 2 previous similar messages [ 4186.617290] Lustre: server umount lustre-MDT0000 complete [ 4190.561790] LustreError: 130483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522427 with bad export cookie 14362214063532103964 [ 4190.569388] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4190.997296] Lustre: server umount lustre-MDT0001 complete [ 4201.075434] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 21:33:57 (1789522437) [ 4209.584388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.394565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.650443] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4215.779498] LustreError: 134873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4215.788194] LustreError: 134873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4220.905137] LustreError: 134872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4223.494149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4223.883269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4227.826769] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4235.159990] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4240.496551] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4244.448626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4244.459563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4244.477035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4248.482780] Lustre: server umount lustre-MDT0000 complete [ 4248.550907] LustreError: 134872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4248.566875] LustreError: 134872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4251.893881] LustreError: 134854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522488 with bad export cookie 14362214063532104846 [ 4251.898848] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4251.917972] LustreError: 134854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4252.200638] Lustre: server umount lustre-MDT0001 complete [ 4260.530773] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 21:34:56 (1789522496) [ 4266.806896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4282.463642] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4287.583599] LustreError: 137288:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 4313.311989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4319.834771] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4329.397275] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4332.261594] Lustre: server umount lustre-OST0000 complete [ 4343.273245] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 21:36:18 (1789522578) [ 4350.478989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4351.153189] LustreError: 138735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4351.242790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4356.418860] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4356.577505] LustreError: 138736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.256961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4370.307459] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4378.080798] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4384.050628] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4391.456729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4391.896508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4391.904220] Lustre: Skipped 1 previous similar message [ 4399.137677] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4409.410409] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4420.582268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4420.591078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4420.606856] Lustre: Skipped 2 previous similar messages [ 4420.629843] Lustre: Skipped 4 previous similar messages [ 4425.580548] Lustre: server umount lustre-MDT0000 complete [ 4425.705132] LustreError: 139463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.716190] LustreError: 139463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4430.158719] LustreError: 138716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522667 with bad export cookie 14362214063532105511 [ 4430.167055] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4430.173198] LustreError: 138716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4430.460341] Lustre: server umount lustre-MDT0001 complete [ 4445.469375] Lustre: server umount lustre-OST0000 complete [ 4456.219062] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 21:38:11 (1789522691) [ 4463.636148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.155814] LustreError: 142389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.225537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4468.194825] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4477.474132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4482.650751] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4490.544304] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4496.900060] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4504.141052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4509.838840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4512.784986] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4520.563548] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4525.038633] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4525.224202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4529.999903] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4530.287500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4532.713784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4532.723265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4532.732075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4532.738138] Lustre: Skipped 2 previous similar messages [ 4532.749180] Lustre: Skipped 1 previous similar message [ 4538.271155] Lustre: server umount lustre-OST0000 complete [ 4542.949206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4556.255375] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4556.532844] Lustre: server umount lustre-MDT0000 complete [ 4558.310876] LustreError: 144393:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.322533] LustreError: 144393:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4560.025710] LustreError: 142369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522797 with bad export cookie 14362214063532106582 [ 4560.034361] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4560.041704] LustreError: 142369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4560.457609] Lustre: server umount lustre-MDT0001 complete [ 4568.685918] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 21:40:04 (1789522804) [ 4574.982693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4590.559875] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4595.681320] LustreError: 146549:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 4621.540142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4621.546558] Lustre: Skipped 2 previous similar messages [ 4628.452404] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4635.962825] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4641.961128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.379031] LustreError: 146574:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.536902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4646.670327] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4653.542456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4658.261525] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4666.931249] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4674.070562] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4679.635712] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4679.904921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4686.293425] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4686.538262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4689.895130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4689.901294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.923659] Lustre: Skipped 1 previous similar message [ 4689.942241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.951211] Lustre: Skipped 8 previous similar messages [ 4703.200096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4703.571661] Lustre: server umount lustre-MDT0000 complete [ 4707.484207] LustreError: 146555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789522944 with bad export cookie 14362214063532107604 [ 4707.504482] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4707.518631] LustreError: 146555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4708.024298] Lustre: server umount lustre-MDT0001 complete [ 4722.489801] Lustre: server umount lustre-OST0000 complete [ 4731.579721] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 21:42:47 (1789522967) [ 4738.418517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4754.015815] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4759.135452] LustreError: 150746:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 4792.318374] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4801.534312] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4809.231775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4809.582422] LustreError: 150771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4809.606029] LustreError: 150771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4809.712552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4814.704431] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4823.558530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4828.863821] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4837.476312] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4843.901214] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4851.336685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4851.543681] Lustre: lustre-OST0001: new disk, initializing [ 4851.548582] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4851.554844] Lustre: 153620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4857.732900] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4858.421499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4858.445943] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4858.611912] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4868.504938] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4873.685736] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4873.963751] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4878.959097] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4879.213474] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4881.376313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4881.391748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.411713] Lustre: Skipped 2 previous similar messages [ 4881.416956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4881.423692] Lustre: Skipped 9 previous similar messages [ 4887.260627] Lustre: server umount lustre-OST0000 complete [ 4894.710894] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4896.673203] Lustre: server umount lustre-OST0001 complete [ 4906.747765] Lustre: server umount lustre-MDT0000 complete [ 4910.387353] LustreError: 150754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523147 with bad export cookie 14362214063532108556 [ 4910.405124] LustreError: 150754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4910.411337] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4910.711333] Lustre: server umount lustre-MDT0001 complete [ 4932.466303] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:46:08 (1789523168) [ 4933.841771] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4935.350655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4936.899341] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:46:12 (1789523172) [ 4942.907044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4943.191470] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4943.221318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4943.431623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4943.436795] Lustre: Skipped 6 previous similar messages [ 4947.637638] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4956.312038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4956.372049] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4956.393611] Lustre: 157181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4960.829310] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4967.692613] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4972.372684] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4978.762422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4979.014716] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4984.645068] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 4986.495926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4991.412418] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4994.712964] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4994.866703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4998.309453] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4998.582169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5001.696738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5001.703232] LustreError: Skipped 1 previous similar message [ 5001.706870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5001.721759] Lustre: Skipped 5 previous similar messages [ 5006.541235] Lustre: server umount lustre-OST0000 complete [ 5027.506230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5027.820536] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5027.831015] Lustre: Skipped 10 previous similar messages [ 5032.952307] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5032.963558] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5033.224600] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5039.409695] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5042.215354] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5042.232439] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5043.417657] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5043.588812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5047.108498] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5047.302615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5051.599968] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9a754437a000.ost_server_uuid 50 [ 5052.714659] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9a754437a000.ost_server_uuid in FULL state after 0 sec [ 5052.847700] LustreError: 157188:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.35@tcp) returned error from blocking AST (req@0000000034d1dc0b x1876450103004800 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000006508b707/0xc750d598b4f37466 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.35@tcp remote: 0x2db222bc38b75710 expref: 5 pid: 157188 timeout: 5152 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5052.875401] LustreError: lustre-MDT0000: A client on nid 192.168.204.35@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5052.881749] LustreError: 157177:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.35@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000006508b707/0xc750d598b4f37466 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.35@tcp remote: 0x2db222bc38b75710 expref: 6 pid: 157188 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5057.505427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5057.522108] LustreError: Skipped 1 previous similar message [ 5062.054437] Lustre: server umount lustre-OST0000 complete [ 5072.380142] Lustre: server umount lustre-MDT0000 complete [ 5072.874548] LustreError: 157187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5072.900836] LustreError: 157187:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 5075.573618] LustreError: 157172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523312 with bad export cookie 14362214063532109739 [ 5075.579719] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5075.585801] LustreError: 157172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5081.921196] Lustre: server umount lustre-MDT0001 complete [ 5090.130193] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5092.336492] Key type lgssc unregistered [ 5092.572831] LNet: 163294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5092.584705] LNetError: 163294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5092.603353] LNet: Removed LNI 192.168.204.135@tcp [ 5093.217828] Key type .llcrypt unregistered [ 5093.221169] Key type ._llcrypt unregistered [ 5111.977177] Key type ._llcrypt registered [ 5111.980075] Key type .llcrypt registered [ 5112.096339] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5112.872296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5112.935332] alg: No test for adler32 (adler32-zlib) [ 5113.916616] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 5114.153691] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5115.871167] Key type lgssc registered [ 5116.964389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5124.210683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5124.230734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.575641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5130.017156] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5135.847418] LustreError: 164680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5135.860896] LustreError: 164680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5137.300483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5137.626144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5140.913559] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5146.932359] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5151.192408] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5157.314165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5157.631337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5163.072640] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5169.900897] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5170.929562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5175.683588] Lustre: Failing over lustre-MDT0000 [ 5175.901912] Lustre: server umount lustre-MDT0000 complete [ 5176.292314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5176.303734] LustreError: 165409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5176.311338] Lustre: Skipped 1 previous similar message [ 5176.330016] LustreError: 165409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5181.413902] LustreError: 164680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5181.438611] LustreError: 164680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5186.531370] LustreError: 164675:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5186.555697] LustreError: 164675:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5187.942232] LustreError: 164661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.35@tcp arrived at 1789523424 with bad export cookie 8122748320054610942 [ 5187.961933] LustreError: 164661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5191.653187] LustreError: 165409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.675644] LustreError: 165409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5192.671382] Lustre: 163991:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789523413/real 1789523413] req@00000000dcfff5f0 x1876451237321984/t0(0) o400->MGC192.168.204.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789523429 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5192.696605] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5194.125197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5202.914996] LustreError: 164681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.950138] LustreError: 164681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 5203.150604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5203.188224] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5206.504613] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5208.299186] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5208.305414] Lustre: lustre-MDT0000: Denying connection for new client 981fb505-fde8-45c1-ad4f-f57ec2cce6b4 (at 192.168.204.35@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5208.562566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5218.794295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5218.803973] Lustre: Skipped 1 previous similar message [ 5218.808842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5218.813261] Lustre: Skipped 1 previous similar message [ 5223.907960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5223.916315] Lustre: Skipped 1 previous similar message [ 5229.024870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.030224] Lustre: Skipped 1 previous similar message [ 5230.047290] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5230.212348] Lustre: server umount lustre-OST0000 complete [ 5233.939205] LustreError: 168411:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5233.947315] Lustre: 167759:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5233.960406] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5233.985609] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5234.000628] LustreError: 167759:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a8dfe99a x1876451237343488/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5234.017058] LustreError: 167759:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5234.028090] LustreError: 167759:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5234.038553] Lustre: lustre-MDT0000: Recovery over after 0:26, of 2 clients 0 recovered and 2 were evicted. [ 5234.148521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5234.160194] Lustre: Skipped 1 previous similar message [ 5234.172870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5234.366753] Lustre: server umount lustre-MDT0000 complete [ 5237.647899] LustreError: 164663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523474 with bad export cookie 8122748320054611040 [ 5237.659628] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5237.662937] LustreError: 164680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5237.664531] LustreError: 164663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5244.066646] Lustre: server umount lustre-MDT0001 complete [ 5253.624624] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5256.171723] Key type lgssc unregistered [ 5256.509688] LNet: 169161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5256.519648] LNetError: 169161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5256.535661] LNet: Removed LNI 192.168.204.135@tcp [ 5257.337265] Key type .llcrypt unregistered [ 5257.339345] Key type ._llcrypt unregistered [ 5277.545791] Key type ._llcrypt registered [ 5277.548762] Key type .llcrypt registered [ 5277.665815] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5278.467895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5278.521106] alg: No test for adler32 (adler32-zlib) [ 5279.602549] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 5279.829464] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5281.495369] Key type lgssc registered [ 5282.373341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5289.252833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5289.278586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.676379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.663706] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5300.203442] LustreError: 170549:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5300.223422] LustreError: 170549:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5302.848825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.228072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5306.905478] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5313.834097] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5318.726242] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5324.355407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.641476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5329.727815] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5335.480097] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5335.925057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5344.234696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5344.241567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.252876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5346.272556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5346.278690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5346.280695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.300780] Lustre: Skipped 1 previous similar message [ 5349.822577] Lustre: server umount lustre-OST0000 complete [ 5354.465150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5354.477199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5354.489401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5356.512880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5356.514210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5359.536993] Lustre: server umount lustre-MDT0000 complete [ 5361.633343] LustreError: 171894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5361.648952] LustreError: 171894:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5362.448540] LustreError: 170532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523599 with bad export cookie 4226837757853147623 [ 5362.461839] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5362.695438] Lustre: server umount lustre-MDT0001 complete [ 5370.382993] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5372.465698] Key type lgssc unregistered [ 5372.732867] LNet: 174079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5372.741918] LNetError: 174079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5373.798741] LNet: Removed LNI 192.168.204.135@tcp [ 5374.511254] Key type .llcrypt unregistered [ 5374.515333] Key type ._llcrypt unregistered [ 5399.547898] Key type ._llcrypt registered [ 5399.550517] Key type .llcrypt registered [ 5399.659627] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5400.444840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5400.466397] alg: No test for adler32 (adler32-zlib) [ 5401.413164] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 5401.600952] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5403.240159] Key type lgssc registered [ 5404.031717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5410.089835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5410.102282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.403891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5414.278979] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5420.312037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.629211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5424.128915] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5429.475755] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5433.614959] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5438.899582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5439.156100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5443.216798] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5448.373591] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5456.557868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5466.085793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5466.090618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5466.102338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5467.106261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5467.106436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5467.116340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.225630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5472.229192] Lustre: Skipped 2 previous similar messages [ 5477.343170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5477.346992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5477.357783] Lustre: Skipped 1 previous similar message [ 5477.473207] Lustre: server umount lustre-OST0000 complete [ 5481.952199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5481.961293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5481.972510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5492.705355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5492.711684] Lustre: Skipped 5 previous similar messages [ 5494.239247] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5494.415963] Lustre: server umount lustre-MDT0000 complete [ 5497.040343] LustreError: 175743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523734 with bad export cookie 1380658012283726404 [ 5497.054656] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.318491] Lustre: server umount lustre-MDT0001 complete [ 5505.808642] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5507.938334] Key type lgssc unregistered [ 5508.166967] LNet: 179319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5508.178151] LNetError: 179319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5508.195931] LNet: Removed LNI 192.168.204.135@tcp [ 5508.879570] Key type .llcrypt unregistered [ 5508.884348] Key type ._llcrypt unregistered [ 5524.164587] Key type ._llcrypt registered [ 5524.166683] Key type .llcrypt registered [ 5524.257767] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5524.947102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5525.006173] alg: No test for adler32 (adler32-zlib) [ 5526.034639] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 5526.221380] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5527.863964] Key type lgssc registered [ 5528.602774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5536.031127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5536.050755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.319135] Lustre: *** cfs_fail_loc=135, val=0*** [ 5537.390588] LustreError: 180818:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5537.398596] LustreError: 180818:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.135@tcp: cfg command failed: rc = -2 [ 5537.406398] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5537.406398] [ 5537.414528] LustreError: MGC192.168.204.135@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5537.429470] LustreError: 180778:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5537.436562] LustreError: 180778:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5537.445111] LustreError: 180778:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5537.457968] LustreError: 180778:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5537.519074] Lustre: server umount lustre-MDT0000 complete [ 5537.522300] LustreError: 180778:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5542.033542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.409097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5545.496119] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5551.557686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5551.871510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5554.631379] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5559.271416] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5563.271323] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5566.433987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.442094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5566.444983] Lustre: Skipped 1 previous similar message [ 5566.457127] Lustre: Skipped 1 previous similar message [ 5570.886834] Lustre: server umount lustre-MDT0000 complete [ 5571.552563] LustreError: 181183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.564599] LustreError: 181183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5573.613710] LustreError: 181169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523810 with bad export cookie 4072610480875234074 [ 5573.624062] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5573.625072] LustreError: 181169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5573.795718] Lustre: server umount lustre-MDT0001 complete [ 5580.418937] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5582.370220] Key type lgssc unregistered [ 5582.554644] LNet: 183506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5582.562706] LNetError: 183506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5582.578196] LNet: Removed LNI 192.168.204.135@tcp [ 5583.137561] Key type .llcrypt unregistered [ 5583.141306] Key type ._llcrypt unregistered [ 5597.298963] Key type ._llcrypt registered [ 5597.301176] Key type .llcrypt registered [ 5597.414365] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5597.624236] Key type .llcrypt unregistered [ 5597.626676] Key type ._llcrypt unregistered [ 5608.514664] Key type ._llcrypt registered [ 5608.517963] Key type .llcrypt registered [ 5608.613702] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5609.299799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5609.330663] alg: No test for adler32 (adler32-zlib) [ 5610.339519] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 5610.514279] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5612.152329] Key type lgssc registered [ 5612.918567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5618.465619] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5618.484610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5634.079412] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5639.199334] LustreError: 185606:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.135@tcp: failed processing log, type 4: rc = -110 [ 5665.823428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5669.532441] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5674.643167] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5679.208065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5679.506910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.551676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5682.538991] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5685.731014] LustreError: 185632:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.754351] LustreError: 185632:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5688.332713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.560753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5691.295301] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5696.950629] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5701.214140] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5703.242667] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5715.424502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5715.434889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.448237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5719.527927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5719.531064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5719.535334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5724.647772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5724.656385] Lustre: Skipped 2 previous similar messages [ 5729.247152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5729.345545] Lustre: server umount lustre-OST0000 complete [ 5734.881361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.883713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5734.883843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5734.889444] Lustre: Skipped 1 previous similar message [ 5734.899407] Lustre: Skipped 1 previous similar message [ 5737.859162] Lustre: server umount lustre-MDT0000 complete [ 5740.001926] LustreError: 186648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.010629] LustreError: 186648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5740.077167] LustreError: 188268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789523977 with bad export cookie 11452809014428409953 [ 5740.079473] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5740.083489] LustreError: 188268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5740.266928] Lustre: server umount lustre-MDT0001 complete [ 5747.404799] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5749.443675] Key type lgssc unregistered [ 5749.641193] LNet: 189666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5749.650696] LNetError: 189666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5749.668580] LNet: Removed LNI 192.168.204.135@tcp [ 5750.299222] Key type .llcrypt unregistered [ 5750.301795] Key type ._llcrypt unregistered [ 5767.546309] Key type ._llcrypt registered [ 5767.548226] Key type .llcrypt registered [ 5767.627400] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5768.412414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5768.425675] alg: No test for adler32 (adler32-zlib) [ 5769.425509] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 5769.632757] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5771.295300] Key type lgssc registered [ 5772.319806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5778.352722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5778.368720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5779.762197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5783.084447] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5789.154828] LustreError: 191054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5789.165881] LustreError: 191054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5789.223308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5789.475344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5792.395096] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5797.405452] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5801.242437] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5805.498355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5805.698543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5809.317046] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5814.234842] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5824.179412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5827.081968] Lustre: Failing over lustre-MDT0000 [ 5827.277853] Lustre: server umount lustre-MDT0000 complete [ 5829.604771] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.607615] LustreError: 191054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5829.621450] Lustre: Skipped 2 previous similar messages [ 5829.638686] LustreError: 191054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5834.720837] LustreError: 191055:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.731317] LustreError: 191055:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5839.840928] LustreError: 192196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.852158] LustreError: 192196:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5842.990748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.098317] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5843.268483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.308128] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5845.339502] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5845.976728] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5848.554734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5848.575178] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5848.606369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5848.769892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5850.114485] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5858.790616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.796476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.802312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.810386] Lustre: Skipped 1 previous similar message [ 5858.824970] Lustre: Skipped 1 previous similar message [ 5863.204803] Lustre: server umount lustre-OST0000 complete [ 5869.031779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.032142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5869.048816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5869.055662] Lustre: Skipped 1 previous similar message [ 5872.689267] Lustre: server umount lustre-MDT0000 complete [ 5874.149101] LustreError: 191054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5874.165027] LustreError: 191054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5875.420773] LustreError: 191036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524112 with bad export cookie 12056910787412418334 [ 5875.422992] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5875.428334] LustreError: 191036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5875.641408] Lustre: server umount lustre-MDT0001 complete [ 5883.257688] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 5885.248564] Key type lgssc unregistered [ 5885.449535] LNet: 195822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5885.457232] LNetError: 195822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5885.476623] LNet: Removed LNI 192.168.204.135@tcp [ 5886.069624] Key type .llcrypt unregistered [ 5886.073714] Key type ._llcrypt unregistered [ 5900.719310] Key type ._llcrypt registered [ 5900.720797] Key type .llcrypt registered [ 5900.795668] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 5901.358161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5901.392435] alg: No test for adler32 (adler32-zlib) [ 5902.363521] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 5902.518817] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 5904.143198] Key type lgssc registered [ 5904.897520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5909.752209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5909.787654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.085927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.667990] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5919.511493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.788436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5922.846916] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5928.111469] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5932.311396] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5936.934717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5937.180789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5941.454547] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 5946.445122] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5956.671585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5959.849363] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5961.638642] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5976.357412] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5985.255958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5985.261423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.284831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5987.297705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5987.298283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5987.311634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.046900] Lustre: server umount lustre-OST0000 complete [ 5996.512578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5996.518590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.528297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5996.532240] Lustre: Skipped 1 previous similar message [ 5999.815447] Lustre: server umount lustre-MDT0000 complete [ 6002.149711] LustreError: 197190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524239 with bad export cookie 10168716220590773047 [ 6002.150934] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6002.162039] LustreError: 197190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6002.378912] Lustre: server umount lustre-MDT0001 complete [ 6009.215575] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6011.159756] Key type lgssc unregistered [ 6011.346852] LNet: 201094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6011.351697] LNetError: 201094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6011.365470] LNet: Removed LNI 192.168.204.135@tcp [ 6011.936121] Key type .llcrypt unregistered [ 6011.938084] Key type ._llcrypt unregistered [ 6022.627917] Key type ._llcrypt registered [ 6022.629205] Key type .llcrypt registered [ 6022.700738] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6023.321952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6023.330678] alg: No test for adler32 (adler32-zlib) [ 6024.268663] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6024.405814] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6026.031744] Key type lgssc registered [ 6026.769746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6031.463313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6031.473232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6032.726330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.194314] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6040.547871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.774710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6043.623674] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6048.358983] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6052.102428] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6056.171200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6056.407903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6057.462505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6060.167500] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6064.899804] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6068.797214] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6074.339246] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6084.044777] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6092.640305] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6106.598576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6106.599243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6106.600376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6106.605912] Lustre: Skipped 1 previous similar message [ 6109.774652] Lustre: server umount lustre-OST0000 complete [ 6112.736825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6112.745164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6112.757837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6112.761389] Lustre: Skipped 1 previous similar message [ 6116.833052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6116.836159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.269257] Lustre: server umount lustre-MDT0000 complete [ 6120.520164] LustreError: 205011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524357 with bad export cookie 12842125584627404015 [ 6120.527301] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.529284] LustreError: 205011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6120.765551] Lustre: server umount lustre-MDT0001 complete [ 6127.170415] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6128.864218] Key type lgssc unregistered [ 6129.047535] LNet: 207498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6129.053829] LNetError: 207498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6129.063051] LNet: Removed LNI 192.168.204.135@tcp [ 6129.568354] Key type .llcrypt unregistered [ 6129.571360] Key type ._llcrypt unregistered [ 6143.601729] Key type ._llcrypt registered [ 6143.603469] Key type .llcrypt registered [ 6143.655444] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6144.113944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6144.179261] alg: No test for adler32 (adler32-zlib) [ 6145.094866] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6145.236908] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6146.847456] Key type lgssc registered [ 6147.535447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6152.115373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6152.126705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.381702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6156.012763] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6160.898862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6161.128250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6163.533703] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6167.654825] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6171.424833] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6175.778045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6176.102339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6178.165510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6180.164818] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6184.975611] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6196.708529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6196.711642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.725433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6196.731370] Lustre: Skipped 1 previous similar message [ 6200.747785] Lustre: server umount lustre-OST0000 complete [ 6206.944845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6206.950337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6206.958616] Lustre: Skipped 2 previous similar messages [ 6209.353552] Lustre: server umount lustre-MDT0000 complete [ 6211.978762] LustreError: 208880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524448 with bad export cookie 10810647700049295166 [ 6211.980340] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6211.990389] LustreError: 208880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6212.221208] Lustre: server umount lustre-MDT0001 complete [ 6218.794166] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6220.516539] Key type lgssc unregistered [ 6220.722646] LNet: 212577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6220.727319] LNetError: 212577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6220.737698] LNet: Removed LNI 192.168.204.135@tcp [ 6221.222128] Key type .llcrypt unregistered [ 6221.224051] Key type ._llcrypt unregistered [ 6231.787754] Key type ._llcrypt registered [ 6231.789425] Key type .llcrypt registered [ 6231.848768] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6232.492081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6232.507671] alg: No test for adler32 (adler32-zlib) [ 6233.421649] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6233.571490] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6235.183230] Key type lgssc registered [ 6235.697543] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6240.156507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6240.182878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.470874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6243.946941] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6248.974029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.244057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6251.522330] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6255.300730] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6258.395230] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6262.028416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6262.232526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.092971] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6266.349895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6269.030609] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6279.649676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6279.655192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.659761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.663537] Lustre: Skipped 1 previous similar message [ 6279.671674] Lustre: Skipped 1 previous similar message [ 6282.741240] Lustre: server umount lustre-OST0000 complete [ 6285.280447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6285.284283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6285.295352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6289.889372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6289.893796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.009970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6295.015955] Lustre: Skipped 2 previous similar messages [ 6299.103129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6299.105815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.119478] Lustre: Skipped 1 previous similar message [ 6299.197670] Lustre: server umount lustre-MDT0000 complete [ 6301.552607] LustreError: 213796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524538 with bad export cookie 4953863396309847762 [ 6301.556356] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6301.559566] LustreError: 213796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6301.762168] Lustre: server umount lustre-MDT0001 complete [ 6307.427784] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6308.944385] Key type lgssc unregistered [ 6309.096963] LNet: 217494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6309.101882] LNetError: 217494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6309.115670] LNet: Removed LNI 192.168.204.135@tcp [ 6309.524210] Key type .llcrypt unregistered [ 6309.526213] Key type ._llcrypt unregistered [ 6319.258036] Key type ._llcrypt registered [ 6319.259653] Key type .llcrypt registered [ 6319.310609] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6319.849651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6319.903474] alg: No test for adler32 (adler32-zlib) [ 6320.841337] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6320.993814] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6322.591284] Key type lgssc registered [ 6323.248487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6327.839835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6327.851834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6329.085698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6331.481759] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6335.868326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.066673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6338.049352] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6341.561908] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6344.221769] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6347.364482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6347.548587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6350.451986] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6354.133902] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6354.730596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6364.238799] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6364.283389] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6364.405783] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6367.569849] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6371.464855] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6372.275986] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6383.156270] Lustre: Permanently deactivating lustre-OST0001 [ 6383.170337] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6390.275167] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6396.976112] Lustre: Permanently reactivating lustre-OST0001 [ 6396.983738] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6396.986937] Lustre: Skipped 2 previous similar messages [ 6403.098577] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.108968] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6403.116957] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6403.125417] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6403.150853] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6406.628558] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.629403] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6406.637905] Lustre: Skipped 2 previous similar messages [ 6406.643287] Lustre: Skipped 1 previous similar message [ 6411.745791] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6411.753446] Lustre: Skipped 1 previous similar message [ 6412.035688] Lustre: server umount lustre-OST0001 complete [ 6416.352504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6416.356837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6416.366656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6416.869113] LustreError: 221581:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.879760] LustreError: 221581:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6421.986213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6421.990368] LustreError: 220554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6421.991405] Lustre: Skipped 2 previous similar messages [ 6422.001316] LustreError: 220554:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6427.106035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6427.108226] LustreError: 220551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6427.111131] Lustre: Skipped 1 previous similar message [ 6428.127276] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6428.245512] Lustre: server umount lustre-OST0000 complete [ 6432.227317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6432.237815] Lustre: Skipped 2 previous similar messages [ 6436.348551] Lustre: server umount lustre-MDT0000 complete [ 6437.344766] LustreError: 218733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6437.351293] LustreError: 218733:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6438.126695] LustreError: 222552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524675 with bad export cookie 16600948198305537377 [ 6438.130962] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6438.136266] LustreError: 222552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6438.277630] Lustre: server umount lustre-MDT0001 complete [ 6443.447301] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6445.162466] Key type lgssc unregistered [ 6445.371670] LNet: 223813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6445.378801] LNetError: 223813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6446.439107] LNet: Removed LNI 192.168.204.135@tcp [ 6446.970666] Key type .llcrypt unregistered [ 6446.973088] Key type ._llcrypt unregistered [ 6467.470885] Key type ._llcrypt registered [ 6467.472775] Key type .llcrypt registered [ 6467.537787] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6468.128522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6468.149755] alg: No test for adler32 (adler32-zlib) [ 6469.057907] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6469.200080] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6470.831457] Key type lgssc registered [ 6471.576796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6475.952302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6475.963896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.143246] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6477.157530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6477.241281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6479.101785] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6483.427401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.485737] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6483.505510] Lustre: 225992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6483.682151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6486.171886] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6490.596418] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6493.524217] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6497.287915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6497.417294] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6497.578851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6501.422305] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6504.492124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6505.187325] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6508.110779] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6516.017909] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6523.862575] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6530.683212] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6545.398513] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6545.403593] Lustre: Skipped 1 previous similar message [ 6570.953748] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6570.957866] Lustre: Skipped 2 previous similar messages [ 6604.873564] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6604.878756] Lustre: Skipped 4 previous similar messages [ 6673.675113] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 6673.678604] Lustre: Skipped 7 previous similar messages [ 6678.497398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6678.498987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6678.501949] LustreError: Skipped 1 previous similar message [ 6678.510165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6678.516549] Lustre: Skipped 1 previous similar message [ 6681.961326] Lustre: server umount lustre-OST0000 complete [ 6688.736110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6688.737388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6688.740488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6688.749696] Lustre: Skipped 2 previous similar messages [ 6689.893389] Lustre: server umount lustre-MDT0000 complete [ 6691.565856] LustreError: 225983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789524928 with bad export cookie 12676194727699471848 [ 6691.566295] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6691.571475] LustreError: 225983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6691.732538] Lustre: server umount lustre-MDT0001 complete [ 6696.480754] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6697.829379] Key type lgssc unregistered [ 6697.971494] LNet: 230783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6697.977329] LNetError: 230783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6697.990161] LNet: Removed LNI 192.168.204.135@tcp [ 6698.402247] Key type .llcrypt unregistered [ 6698.404265] Key type ._llcrypt unregistered [ 6709.912827] Key type ._llcrypt registered [ 6709.914422] Key type .llcrypt registered [ 6709.961833] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6710.530613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6710.545255] alg: No test for adler32 (adler32-zlib) [ 6711.446167] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6711.583517] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6713.183197] Key type lgssc registered [ 6713.696887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6717.639151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6717.648818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.901377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6720.885622] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6725.115496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6725.323113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6727.514966] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6730.926883] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6733.391783] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6736.456688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6736.653704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6739.391430] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_default_debug -1 all [ 6743.138156] Lustre: DEBUG MARKER: oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6743.851334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 6756.323706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6756.324430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6756.329409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6756.334548] Lustre: Skipped 1 previous similar message [ 6756.340894] Lustre: Skipped 1 previous similar message [ 6757.409128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6757.412389] Lustre: Skipped 1 previous similar message [ 6758.120127] Lustre: server umount lustre-OST0000 complete [ 6761.440056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6761.443563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.449179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6761.953499] Lustre: server umount lustre-MDT0000 complete [ 6762.464234] LustreError: 232171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6762.464702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6762.472458] LustreError: 232171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6763.439285] LustreError: 232153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789525000 with bad export cookie 11772989679823497800 [ 6763.441286] LustreError: MGC192.168.204.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6763.444804] LustreError: 232153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6763.579945] Lustre: server umount lustre-MDT0001 complete [ 6768.099364] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 6769.386677] Key type lgssc unregistered [ 6769.534644] LNet: 235861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6769.538175] LNetError: 235861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6769.548679] LNet: Removed LNI 192.168.204.135@tcp [ 6769.920241] Key type .llcrypt unregistered [ 6769.922764] Key type ._llcrypt unregistered [ 6780.148572] Key type ._llcrypt registered [ 6780.150136] Key type .llcrypt registered [ 6780.196933] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 6780.603697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6780.674311] alg: No test for adler32 (adler32-zlib) [ 6781.576938] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 6781.716174] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 6783.327260] Key type lgssc registered [ 6783.959662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7049.083292] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7050.253885] Key type lgssc unregistered [ 7050.406582] LNet: 237854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7050.410245] LNetError: 237854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7050.417515] LNet: Removed LNI 192.168.204.135@tcp [ 7050.860860] Key type .llcrypt unregistered [ 7050.862704] Key type ._llcrypt unregistered [ 7061.864314] Key type ._llcrypt registered [ 7061.865524] Key type .llcrypt registered [ 7061.913300] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing set_hostid [ 7069.360867] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing load_modules_local [ 7069.718742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7069.741098] alg: No test for adler32 (adler32-zlib) [ 7070.629181] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 7070.738712] LNet: Added LNI 192.168.204.135@tcp [8/256/0/180] [ 7072.335255] Key type lgssc registered [ 7072.775430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7084.845687] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6819 sec ========== 22:22:01 (1789525321) [ 7085.549812] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:22:02 (1789525322) === [ 7086.917582] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:22:03 (1789525323) === [ 7101.459929] Lustre: DEBUG MARKER: oleg435-server.virtnet: executing unload_modules_local [ 7102.567325] Key type lgssc unregistered [ 7102.703421] LNet: 243056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7102.706386] LNetError: 243056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7102.713268] LNet: Removed LNI 192.168.204.135@tcp [ 7103.031446] Key type .llcrypt unregistered [ 7103.032540] Key type ._llcrypt unregistered