[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496820098 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003138] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005027] kvm-guest: setup PV IPIs [ 0.008656] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011136] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022265] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026639] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027139] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] *** VALIDATE tmpfs *** [ 0.031408] *** VALIDATE proc *** [ 0.032159] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.035031] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041963] debug: unmapping init [mem 0xffffffff9dc59000-0xffffffff9dc60fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044645] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045029] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052332] rcu: Hierarchical SRCU implementation. [ 0.054497] smp: Bringing up secondary CPUs ... [ 0.055619] x86: Booting SMP configuration: [ 0.056035] .... node #0, CPUs: #1 #2 #3 [ 0.059153] smp: Brought up 1 node, 4 CPUs [ 0.061022] smpboot: Max logical packages: 1 [ 0.062020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.102653] node 0 deferred pages initialised in 38ms [ 0.105078] devtmpfs: initialized [ 0.105855] x86/mm: Memory block size: 128MB [ 0.108232] gcov: version magic: 0x41383552 [ 0.110143] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.111000] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.112260] pinctrl core: initialized pinctrl subsystem [ 0.113132] [ 0.113570] ************************************************************* [ 0.114011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.115009] ** ** [ 0.116011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.118009] ** ** [ 0.120010] ** This means that this kernel is built to expose internal ** [ 0.122012] ** IOMMU data structures, which may compromise security on ** [ 0.124012] ** your system. ** [ 0.126012] ** ** [ 0.128013] ** If you see this message and you are not debugging the ** [ 0.130012] ** kernel, report this immediately to your vendor! ** [ 0.131009] ** ** [ 0.133010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134010] ************************************************************* [ 0.136531] NET: Registered protocol family 16 [ 0.138305] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.139053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.141083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.145089] cpuidle: using governor menu [ 0.146483] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.147421] PCI: Using configuration type 1 for base access [ 0.148128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.155340] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.158029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.162073] cryptd: max_cpu_qlen set to 1000 [ 0.163251] ACPI: Added _OSI(Module Device) [ 0.165014] ACPI: Added _OSI(Processor Device) [ 0.166013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.168015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.173397] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.180557] ACPI: Interpreter enabled [ 0.181057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.183010] ACPI: Using IOAPIC for interrupt routing [ 0.184106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.187400] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.194960] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.197061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.200020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.204109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.209618] acpiphp: Slot [2] registered [ 0.211167] acpiphp: Slot [5] registered [ 0.213164] acpiphp: Slot [6] registered [ 0.214109] acpiphp: Slot [7] registered [ 0.216205] acpiphp: Slot [8] registered [ 0.217126] acpiphp: Slot [9] registered [ 0.219110] acpiphp: Slot [10] registered [ 0.221143] acpiphp: Slot [3] registered [ 0.222145] acpiphp: Slot [4] registered [ 0.224117] acpiphp: Slot [11] registered [ 0.225127] acpiphp: Slot [12] registered [ 0.227104] acpiphp: Slot [13] registered [ 0.228108] acpiphp: Slot [14] registered [ 0.230097] acpiphp: Slot [15] registered [ 0.231103] acpiphp: Slot [16] registered [ 0.233112] acpiphp: Slot [17] registered [ 0.234096] acpiphp: Slot [18] registered [ 0.236139] acpiphp: Slot [19] registered [ 0.237110] acpiphp: Slot [20] registered [ 0.239114] acpiphp: Slot [21] registered [ 0.240101] acpiphp: Slot [22] registered [ 0.242105] acpiphp: Slot [23] registered [ 0.243116] acpiphp: Slot [24] registered [ 0.245122] acpiphp: Slot [25] registered [ 0.246119] acpiphp: Slot [26] registered [ 0.248116] acpiphp: Slot [27] registered [ 0.250099] acpiphp: Slot [28] registered [ 0.251548] acpiphp: Slot [29] registered [ 0.253100] acpiphp: Slot [30] registered [ 0.255101] acpiphp: Slot [31] registered [ 0.256067] PCI host bridge to bus 0000:00 [ 0.258021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.260025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.262026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.265038] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.268023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.270024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.272168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.275000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.278304] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.291996] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.297054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.299019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.302019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.305019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.308540] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.310893] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.315066] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.319898] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.325025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.339041] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.345024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.350732] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.362019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.371022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.393020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.404846] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.411028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.419032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.434025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.443998] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.472025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.489017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.502694] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.512027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.520025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.537031] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.546389] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.559018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.567026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.590020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.599728] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.606019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.611025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.623025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.632358] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.634397] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.637441] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.639435] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.642296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.647150] iommu: Default domain type: Passthrough [ 0.649459] SCSI subsystem initialized [ 0.651154] ACPI: bus type USB registered [ 0.652143] usbcore: registered new interface driver usbfs [ 0.654229] usbcore: registered new interface driver hub [ 0.656085] usbcore: registered new device driver usb [ 0.658201] pps_core: LinuxPPS API ver. 1 registered [ 0.660015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.663060] PTP clock support registered [ 0.665107] EDAC MC: Ver: 3.0.0 [ 0.667198] PCI: Using ACPI for IRQ routing [ 0.670058] NetLabel: Initializing [ 0.671030] NetLabel: domain hash size = 128 [ 0.672017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.675120] NetLabel: unlabeled traffic allowed by default [ 0.678060] vgaarb: loaded [ 0.679316] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.681017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.688136] clocksource: Switched to clocksource kvm-clock [ 0.797613] VFS: Disk quotas dquot_6.6.0 [ 0.799277] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.801728] *** VALIDATE ramfs *** [ 0.803109] *** VALIDATE hugetlbfs *** [ 0.804671] pnp: PnP ACPI init [ 0.806982] pnp: PnP ACPI: found 6 devices [ 0.831132] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.834264] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.836376] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.838461] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.840805] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.843130] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.845704] NET: Registered protocol family 2 [ 0.848138] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.852753] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.856227] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.861298] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.864587] TCP: Hash tables configured (established 65536 bind 65536) [ 0.867411] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.870314] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.872896] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.875752] NET: Registered protocol family 1 [ 0.878078] RPC: Registered named UNIX socket transport module. [ 0.880089] RPC: Registered udp transport module. [ 0.881597] RPC: Registered tcp transport module. [ 0.883153] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.885422] NET: Registered protocol family 44 [ 0.889326] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.891136] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.893152] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.895354] PCI: CLS 0 bytes, default 64 [ 0.897130] Unpacking initramfs... [ 2.354708] debug: unmapping init [mem 0xffff9c91bcc54000-0xffff9c91bffbffff] [ 2.361307] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.363365] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.367536] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.870546] Initialise system trusted keyrings [ 2.872239] Key type blacklist registered [ 2.874125] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.883238] zbud: loaded [ 2.886243] *** VALIDATE nfs *** [ 2.887495] *** VALIDATE nfs4 *** [ 2.889100] pstore: using deflate compression [ 2.893307] Platform Keyring initialized [ 2.989484] NET: Registered protocol family 38 [ 2.991502] Key type asymmetric registered [ 2.993216] Asymmetric key parser 'x509' registered [ 2.995347] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.999104] io scheduler mq-deadline registered [ 3.000832] io scheduler kyber registered [ 3.002517] io scheduler bfq registered [ 3.004374] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.007468] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.010561] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.013350] ACPI: Power Button [PWRF] [ 3.018934] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.026271] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.050421] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.062784] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.096484] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.127383] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.157351] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.161679] Non-volatile memory driver v1.3 [ 3.163551] Linux agpgart interface v0.103 [ 3.195891] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.199297] vda: detected capacity change from 0 to 77115392 [ 3.225732] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.229343] vdb: detected capacity change from 0 to 1073741824 [ 3.258991] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.262237] vdc: detected capacity change from 0 to 2621440000 [ 3.277188] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.280040] vdd: detected capacity change from 0 to 2621440000 [ 3.303215] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.306477] vde: detected capacity change from 0 to 4294967296 [ 3.340299] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.343346] vdf: detected capacity change from 0 to 4294967296 [ 3.360420] libphy: Fixed MDIO Bus: probed [ 3.370278] usbcore: registered new interface driver usbserial_generic [ 3.372995] usbserial: USB Serial support registered for generic [ 3.375716] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.380370] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.382470] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.385248] mousedev: PS/2 mouse device common for all mice [ 3.388775] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.390555] rtc_cmos 00:05: RTC can wake from S4 [ 3.395247] rtc_cmos 00:05: registered as rtc0 [ 3.395592] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.396541] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.402196] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.403333] intel_pstate: CPU model not supported [ 3.408371] hid: raw HID events driver (C) Jiri Kosina [ 3.411340] usbcore: registered new interface driver usbhid [ 3.413472] usbhid: USB HID core driver [ 3.415216] drop_monitor: Initializing network drop monitor service [ 3.417869] Initializing XFRM netlink socket [ 3.419680] NET: Registered protocol family 10 [ 3.423147] Segment Routing with IPv6 [ 3.424227] NET: Registered protocol family 17 [ 3.425554] mpls_gso: MPLS GSO support [ 3.431837] RAS: Correctable Errors collector initialized. [ 3.436268] AVX version of gcm_enc/dec engaged. [ 3.437929] AES CTR mode by8 optimization enabled [ 3.523183] sched_clock: Marking stable (3523135742, 0)->(4426854714, -903718972) [ 3.526310] registered taskstats version 1 [ 3.528476] Loading compiled-in X.509 certificates [ 3.530194] zswap: loaded using pool lzo/zbud [ 3.555771] Key type big_key registered [ 3.567797] Key type encrypted registered [ 3.569508] ima: No TPM chip found, activating TPM-bypass! [ 3.571364] ima: Allocated hash algorithm: sha1 [ 3.573056] ima: No architecture policies found [ 3.574844] evm: Initialising EVM extended attributes: [ 3.576703] evm: security.selinux [ 3.577884] evm: security.ima [ 3.578954] evm: security.capability [ 3.580205] evm: HMAC attrs: 0x1 [ 3.582543] rtc_cmos 00:05: setting system clock to 2026-10-02 19:49:53 UTC (1790970593) [ 3.588648] debug: unmapping init [mem 0xffffffff9ec03000-0xffffffff9edfffff] [ 3.591582] debug: unmapping init [mem 0xffffffff9d982000-0xffffffff9dc58fff] [ 3.599221] Write protecting the kernel read-only data: 28672k [ 3.602018] debug: unmapping init [mem 0xffffffff9c003000-0xffffffff9c1fffff] [ 3.604595] debug: unmapping init [mem 0xffffffff9c914000-0xffffffff9c9fffff] [ 3.639775] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.647646] systemd[1]: Detected virtualization kvm. [ 3.649586] systemd[1]: Detected architecture x86-64. [ 3.651395] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.677150] systemd[1]: No hostname configured. [ 3.678467] systemd[1]: Set hostname to . [ 3.680526] random: systemd: uninitialized urandom read (16 bytes read) [ 3.682727] systemd[1]: Initializing machine ID from random generator. [ 3.721129] random: ln: uninitialized urandom read (6 bytes read) [ 3.822517] random: systemd: uninitialized urandom read (16 bytes read) [ 3.825624] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.831195] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.837127] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.478137] device-mapper: uevent: version 1.0.3 [ 4.480477] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.231358] random: fast init done [ 5.235223] virtio_net virtio0 ens2: renamed from eth0 [ 5.297347] scsi host0: ata_piix [ 5.300834] scsi host1: ata_piix [ 5.302670] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.305289] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.764245] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.088080] random: crng init done [ 10.089445] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.498195] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.648388] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.880858] SELinux: Disabled at runtime. [ 11.934861] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.942795] systemd[1]: Detected virtualization kvm. [ 11.944931] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.392137] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.395209] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.399628] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.405336] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.408537] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.415485] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.419678] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ 12.460919] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.812599] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.128430] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.132481] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.193620] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.205640] EDAC sbridge: Ver: 1.1.2 [ 14.704057] Key type dns_resolver registered [ 14.976599] NFS: Registering the id_resolver key type [ 14.978444] Key type id_resolver registered [ 14.980027] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg133-server login: [ 55.819162] hrtimer: interrupt took 4143651 ns [ 63.723272] libcfs: loading out-of-tree module taints kernel. [ 63.789641] Key type ._llcrypt registered [ 63.791983] Key type .llcrypt registered [ 63.966311] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 81.633514] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 83.134382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.153375] alg: No test for adler32 (adler32-zlib) [ 84.508771] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 85.360852] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 87.111267] Key type lgssc registered [ 88.451247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.682263] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.277339] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 160.722143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 160.751647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.924394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.951097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 162.032837] Lustre: lustre-MDT0000: new disk, initializing [ 162.109725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 162.126646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 166.704883] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.980450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.088174] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 180.121824] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 180.127276] Lustre: Skipped 1 previous similar message [ 180.235747] Lustre: lustre-MDT0001: new disk, initializing [ 180.296770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.339214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.348555] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 185.341394] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.117957] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 200.850280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.298832] Lustre: lustre-OST0000: new disk, initializing [ 201.304879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.312073] Lustre: 8449:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.441860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.794893] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.965090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.978695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 209.074674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.853718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.967819] Lustre: lustre-OST0001: new disk, initializing [ 221.970774] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.975296] Lustre: 9521:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.043619] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.968635] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.966764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.979040] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.116422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 240.365814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 246.352423] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.583888] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing check_logdir /tmp/testlogs/ [ 257.837253] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing yml_node [ 262.059223] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 264.865182] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 268.019967] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 269.688326] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Oct 2 15:54:17 EDT 2026 [ 286.781806] Lustre: DEBUG MARKER: only running tests: 0-31 [ 288.122919] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 289.698674] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 308.191710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 308.195429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.201920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.220499] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.220978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.255144] Lustre: Skipped 2 previous similar messages [ 309.279164] Lustre: Skipped 3 previous similar messages [ 311.488059] Lustre: server umount lustre-MDT0000 complete [ 316.229354] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790970906 with bad export cookie 15128805676280554933 [ 316.236341] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.251551] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 319.459440] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.460723] LustreError: 7911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.472285] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.480054] LustreError: 7911:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 319.484711] Lustre: Skipped 1 previous similar message [ 321.504874] LustreError: 6515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.511627] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.532621] LustreError: 6515:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 321.561139] Lustre: Skipped 1 previous similar message [ 322.641937] Lustre: server umount lustre-MDT0001 complete [ 326.530904] Lustre: server umount lustre-OST0000 complete [ 330.520928] Lustre: server umount lustre-OST0001 complete [ 337.122983] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 347.282186] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 385.024487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.224773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 385.254110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 385.320592] Lustre: lustre-MDT0000: new disk, initializing [ 385.432369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 385.454397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 390.219979] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 401.023676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.198748] Lustre: 15119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 401.262854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 401.268740] Lustre: Skipped 1 previous similar message [ 401.358311] Lustre: lustre-MDT0001: new disk, initializing [ 401.444525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 401.492525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 401.502248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 406.121610] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 415.991197] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 422.056849] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 429.723039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.978213] Lustre: lustre-OST0000: new disk, initializing [ 429.983118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.990343] Lustre: 17090:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 430.084166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.001308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 432.010256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 432.113832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 436.933633] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 447.910917] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 453.267695] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 453.469710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 458.357955] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 458.640094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 462.823258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.838057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.857036] Lustre: Skipped 1 previous similar message [ 462.873902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.922105] Lustre: server umount lustre-OST0000 complete [ 473.058829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 473.060563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.071791] LustreError: Skipped 1 previous similar message [ 473.073268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 473.089730] Lustre: Skipped 2 previous similar messages [ 473.099832] Lustre: Skipped 2 previous similar messages [ 485.343175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 485.499567] Lustre: server umount lustre-MDT0000 complete [ 487.395648] LustreError: 15923:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.419097] LustreError: 15923:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 489.954410] LustreError: 17089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971079 with bad export cookie 15128805676280556466 [ 489.958831] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.964310] LustreError: 17089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 490.486566] Lustre: server umount lustre-MDT0001 complete [ 495.211252] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 15:58:03 (1790971083) [ 503.340888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.851958] LustreError: 19357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.908620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.879689] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 508.898747] LustreError: 19358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.140409] LustreError: 19357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.162134] LustreError: 19357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 519.214950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 525.207736] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 534.769312] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 540.591428] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 547.903538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 554.321447] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 561.591506] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 573.924058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 573.930043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.941091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.944335] Lustre: Skipped 4 previous similar messages [ 577.536694] Lustre: server umount lustre-OST0000 complete [ 587.478364] Lustre: server umount lustre-MDT0000 complete [ 590.310638] LustreError: 20301:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.322031] LustreError: 20301:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 591.375552] LustreError: 19339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971181 with bad export cookie 15128805676280557348 [ 591.382365] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.393104] LustreError: 19339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 592.037085] Lustre: server umount lustre-MDT0001 complete [ 604.665774] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 607.920780] Key type lgssc unregistered [ 608.367748] LNet: 22888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.376466] LNetError: 22888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.447973] LNet: Removed LNI 192.168.201.133@tcp [ 610.558634] Key type .llcrypt unregistered [ 610.565109] Key type ._llcrypt unregistered [ 636.218895] Key type ._llcrypt registered [ 636.222612] Key type .llcrypt registered [ 636.369900] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 637.927301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 638.288273] alg: No test for adler32 (adler32-zlib) [ 639.696205] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 639.893915] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 641.607198] Key type lgssc registered [ 642.445949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.549128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 652.571552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.218810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.877130] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 664.549333] LustreError: 24316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.556207] LustreError: 24316:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 668.278652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.764941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 675.332759] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 686.316887] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 692.410236] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 699.912932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 700.423397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.710327] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 715.089744] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 725.048485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 734.178321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 734.195802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.216663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 735.213903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 735.217178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.239186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 737.888889] Lustre: server umount lustre-OST0000 complete [ 745.440748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 745.444911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 745.449310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 745.451456] Lustre: Skipped 1 previous similar message [ 745.458475] Lustre: Skipped 2 previous similar messages [ 747.942323] Lustre: server umount lustre-MDT0000 complete [ 750.566947] LustreError: 24317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 750.593761] LustreError: 24317:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 751.842300] LustreError: 24298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971341 with bad export cookie 6250876119297940877 [ 751.854650] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 751.857467] LustreError: 24298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 752.310589] Lustre: server umount lustre-MDT0001 complete [ 763.409450] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 765.957733] Key type lgssc unregistered [ 766.225670] LNet: 28310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.243228] LNetError: 28310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.285788] LNet: Removed LNI 192.168.201.133@tcp [ 768.297868] Key type .llcrypt unregistered [ 768.301122] Key type ._llcrypt unregistered [ 794.062733] Key type ._llcrypt registered [ 794.065743] Key type .llcrypt registered [ 794.190764] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 795.122218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 795.213635] alg: No test for adler32 (adler32-zlib) [ 796.395326] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 796.827423] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 798.623248] Key type lgssc registered [ 800.416456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 809.315385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 809.340847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.765683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.812807] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 820.202395] LustreError: 29742:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.231604] LustreError: 29742:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 825.100610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.429053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 830.280813] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 837.839543] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 844.080864] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 859.211739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 859.716948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.027957] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 876.049531] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 880.189258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 890.342669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 890.359829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.366791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 890.372398] Lustre: Skipped 1 previous similar message [ 893.686225] Lustre: server umount lustre-OST0000 complete [ 900.580167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.592142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 900.599761] Lustre: Skipped 1 previous similar message [ 900.617127] Lustre: Skipped 2 previous similar messages [ 903.986198] Lustre: server umount lustre-MDT0000 complete [ 905.696771] LustreError: 29736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.714149] LustreError: 29736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 908.009669] LustreError: 29721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971497 with bad export cookie 2066084306526630402 [ 908.020587] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 908.035483] LustreError: 29721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 908.656956] Lustre: server umount lustre-MDT0001 complete [ 921.114825] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 923.802429] Key type lgssc unregistered [ 924.079712] LNet: 33620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.084597] LNetError: 33620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.101586] LNet: Removed LNI 192.168.201.133@tcp [ 925.084261] Key type .llcrypt unregistered [ 925.087615] Key type ._llcrypt unregistered [ 950.282909] Key type ._llcrypt registered [ 950.286101] Key type .llcrypt registered [ 950.440315] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 951.697686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 951.729272] alg: No test for adler32 (adler32-zlib) [ 952.995178] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 953.427111] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 955.217097] Key type lgssc registered [ 956.171758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 965.357873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 965.384053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.010273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.980524] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 976.358949] LustreError: 35049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.382482] LustreError: 35049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 981.472956] LustreError: 35048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 983.835786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.082865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 988.934587] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 997.589956] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1004.359615] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1012.040612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.514548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.013318] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1028.175861] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1036.157205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1049.574190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1049.584888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.610391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.619137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1051.626743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.654166] Lustre: server umount lustre-OST0000 complete [ 1060.837733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1060.851419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.866414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1060.873945] Lustre: Skipped 1 previous similar message [ 1064.174858] Lustre: server umount lustre-MDT0000 complete [ 1066.980225] LustreError: 35049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1067.009254] LustreError: 35049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1069.119918] LustreError: 35029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971659 with bad export cookie 2997207452882546437 [ 1069.146448] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1069.701247] Lustre: server umount lustre-MDT0001 complete [ 1083.223246] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1086.914395] Key type lgssc unregistered [ 1087.319848] LNet: 38602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.328174] LNetError: 38602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.358293] LNet: Removed LNI 192.168.201.133@tcp [ 1088.491256] Key type .llcrypt unregistered [ 1088.493589] Key type ._llcrypt unregistered [ 1112.882527] Key type ._llcrypt registered [ 1112.886503] Key type .llcrypt registered [ 1113.019706] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1114.256820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1114.361164] alg: No test for adler32 (adler32-zlib) [ 1115.580852] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1115.808560] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1117.511222] Key type lgssc registered [ 1118.849909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1128.216887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1128.245456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1130.020577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1136.223412] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1139.191927] LustreError: 40031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.231136] LustreError: 40031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1144.297981] LustreError: 40030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.216386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.685872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1154.516839] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1163.975663] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1170.971341] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1178.953377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1179.447220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.011645] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1196.005613] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1199.926160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1207.087827] Lustre: lustre-OST0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1209.313289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1209.322650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.335731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1210.343402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1210.356500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1210.358079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.390267] Lustre: Skipped 1 previous similar message [ 1212.155151] Lustre: server umount lustre-OST0000 complete [ 1235.949016] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1235.958601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.964414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.983040] Lustre: Skipped 1 previous similar message [ 1240.415610] Lustre: server umount lustre-MDT0000 complete [ 1242.102093] LustreError: 40759:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.116905] LustreError: 40759:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1245.022737] LustreError: 40012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971834 with bad export cookie 16018498292293336759 [ 1245.030977] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.037565] LustreError: 40012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1251.733037] Lustre: server umount lustre-MDT0001 complete [ 1265.066757] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1268.418904] Key type lgssc unregistered [ 1268.824708] LNet: 43744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.832186] LNetError: 43744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.865426] LNet: Removed LNI 192.168.201.133@tcp [ 1270.040853] Key type .llcrypt unregistered [ 1270.044952] Key type ._llcrypt unregistered [ 1296.706370] Key type ._llcrypt registered [ 1296.707952] Key type .llcrypt registered [ 1296.801078] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1297.332573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1297.428692] alg: No test for adler32 (adler32-zlib) [ 1298.446673] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1298.666582] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1300.335260] Key type lgssc registered [ 1301.839155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1311.595207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1311.639816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1313.675556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.026770] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1324.008359] LustreError: 45170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.030610] LustreError: 45170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1328.119924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.483974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1333.261313] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1340.755277] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1347.871338] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1355.518132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1355.896796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.964364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1362.917083] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1370.815611] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1379.827386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1379.837762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.856070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1383.209534] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1383.920448] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1383.951422] Lustre: Skipped 1 previous similar message [ 1386.380725] Lustre: server umount lustre-MDT0000 complete [ 1388.297672] LustreError: 45167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.834844] LustreError: 45153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790971981 with bad export cookie 15653338499999174357 [ 1391.852794] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1391.858114] LustreError: 45153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1393.387088] LustreError: 46395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.402728] LustreError: 46395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1393.426741] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (stopping) [ 1393.436386] Lustre: Skipped 3 previous similar messages [ 1394.145736] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.195280] LustreError: 46395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1396.195282] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1396.195294] Lustre: Skipped 1 previous similar message [ 1396.212373] LustreError: 46395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1398.321113] Lustre: server umount lustre-MDT0001 complete [ 1413.196839] Lustre: server umount lustre-OST0000 complete [ 1430.461387] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1433.595329] Key type lgssc unregistered [ 1434.030097] LNet: 49033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.036375] LNetError: 49033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.049335] LNet: Removed LNI 192.168.201.133@tcp [ 1435.084222] Key type .llcrypt unregistered [ 1435.094643] Key type ._llcrypt unregistered [ 1463.608470] Key type ._llcrypt registered [ 1463.610865] Key type .llcrypt registered [ 1463.720858] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1465.592720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1465.637793] alg: No test for adler32 (adler32-zlib) [ 1466.802577] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1467.204649] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1469.023361] Key type lgssc registered [ 1470.663933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1480.084151] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1480.111069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1495.647452] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1500.767779] LustreError: 50434:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 1527.519418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1535.550930] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1544.716783] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1713.035644] Lustre: server umount lustre-OST0000 complete [ 1728.974381] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1731.417970] Key type lgssc unregistered [ 1731.792893] LNet: 52120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.803370] LNetError: 52120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.838745] LNet: Removed LNI 192.168.201.133@tcp [ 1732.611657] Key type .llcrypt unregistered [ 1732.614121] Key type ._llcrypt unregistered [ 1756.194793] Key type ._llcrypt registered [ 1756.196793] Key type .llcrypt registered [ 1756.283343] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1757.446457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1757.653698] alg: No test for adler32 (adler32-zlib) [ 1758.777602] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1759.032962] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1760.663356] Key type lgssc registered [ 1762.276384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1770.911783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1770.974173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1772.712931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.138178] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1782.252748] LustreError: 53550:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1782.283051] LustreError: 53550:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1786.231827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.639166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1790.447453] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1797.210660] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1802.590524] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1808.278460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1808.581568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1813.880797] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1817.792903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1820.963097] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1825.457857] Lustre: server umount lustre-OST0000 complete [ 1832.932732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1832.935663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.941097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.441374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.442260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1833.468149] Lustre: Skipped 1 previous similar message [ 1838.562090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1838.568266] Lustre: Skipped 1 previous similar message [ 1839.670858] Lustre: server umount lustre-MDT0000 complete [ 1843.117795] LustreError: 53532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972433 with bad export cookie 6855547564684770158 [ 1843.129234] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1843.131956] LustreError: 53532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1853.606352] Lustre: server umount lustre-MDT0001 complete [ 1867.046047] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1869.856759] Key type lgssc unregistered [ 1870.122689] LNet: 57083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.133182] LNetError: 57083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.205742] LNet: Removed LNI 192.168.201.133@tcp [ 1871.901163] Key type .llcrypt unregistered [ 1871.904648] Key type ._llcrypt unregistered [ 1895.338454] Key type ._llcrypt registered [ 1895.341713] Key type .llcrypt registered [ 1895.474708] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1896.577444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1896.671348] alg: No test for adler32 (adler32-zlib) [ 1897.776481] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 1897.999617] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1899.696483] Key type lgssc registered [ 1900.776477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1908.017920] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 1908.058850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1923.615800] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1928.735499] LustreError: 58488:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 1955.423384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.873868] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1971.039192] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1979.134418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.087143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.207948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1985.507198] LustreError: 58514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.548381] LustreError: 58514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1985.656498] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1990.630207] LustreError: 58513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.657083] LustreError: 58514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.664415] LustreError: 58514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1995.108601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.560295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2000.586483] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2008.619362] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2014.544024] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2020.832688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2020.839680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.865821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2021.361586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2021.372358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.385137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2021.387825] Lustre: Skipped 1 previous similar message [ 2023.162061] Lustre: server umount lustre-OST0000 complete [ 2045.537699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.540450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.554649] Lustre: Skipped 1 previous similar message [ 2051.961692] Lustre: server umount lustre-MDT0000 complete [ 2052.084179] LustreError: 59530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.110989] LustreError: 59530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2055.579220] LustreError: 58496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972645 with bad export cookie 13109914122196921207 [ 2055.582660] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2055.587060] LustreError: 58496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2062.064859] Lustre: server umount lustre-MDT0001 complete [ 2073.431880] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2075.914320] Key type lgssc unregistered [ 2076.176485] LNet: 62230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.189858] LNetError: 62230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.210632] LNet: Removed LNI 192.168.201.133@tcp [ 2077.096620] Key type .llcrypt unregistered [ 2077.098548] Key type ._llcrypt unregistered [ 2102.743561] Key type ._llcrypt registered [ 2102.745251] Key type .llcrypt registered [ 2102.873650] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2104.179931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2104.347238] alg: No test for adler32 (adler32-zlib) [ 2105.488498] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2105.874161] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2107.519503] Key type lgssc registered [ 2108.939315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2117.282917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2117.327212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.897496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.568632] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2128.374053] LustreError: 63661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.404504] LustreError: 63661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2131.972928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.318580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2137.461697] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2146.831588] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2153.877086] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2161.308492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2161.711833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2163.796249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2168.644699] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2176.644779] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2203.105425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.112726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.127497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2203.143600] Lustre: Skipped 1 previous similar message [ 2203.164287] Lustre: Skipped 1 previous similar message [ 2208.231597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2208.238678] Lustre: Skipped 1 previous similar message [ 2213.353360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2213.363110] Lustre: Skipped 1 previous similar message [ 2215.391323] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2215.577592] Lustre: server umount lustre-OST0000 complete [ 2219.489478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2219.495451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.520365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.584111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.587068] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.935198] Lustre: server umount lustre-MDT0000 complete [ 2228.716737] LustreError: 63661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.732028] LustreError: 63661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2229.716583] LustreError: 63644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790972819 with bad export cookie 18047356891182749058 [ 2229.721584] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2229.728184] LustreError: 63644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2230.269744] Lustre: server umount lustre-MDT0001 complete [ 2241.015437] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2242.968452] Key type lgssc unregistered [ 2243.255840] LNet: 67220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.261947] LNetError: 67220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2243.283757] LNet: Removed LNI 192.168.201.133@tcp [ 2244.273372] Key type .llcrypt unregistered [ 2244.284969] Key type ._llcrypt unregistered [ 2270.018263] Key type ._llcrypt registered [ 2270.019962] Key type .llcrypt registered [ 2270.124557] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2271.143039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.220614] alg: No test for adler32 (adler32-zlib) [ 2272.422130] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2272.610067] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2274.329143] Key type lgssc registered [ 2275.869617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2286.126963] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 16:27:54 (1790972874) [ 2293.586926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2293.631121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2295.382406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2300.161678] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2305.504364] LustreError: 68940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.521762] LustreError: 68940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2309.978236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.496505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2315.375691] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2324.167665] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2330.340312] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2337.842674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2338.367150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.440778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2344.545980] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2350.852416] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2360.894098] Lustre: Failing over lustre-MDT0000 [ 2361.243103] Lustre: server umount lustre-MDT0000 complete [ 2361.323432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.345597] LustreError: 68935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.366684] Lustre: Skipped 2 previous similar messages [ 2363.133355] LustreError: 68937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.149246] LustreError: 68937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2366.438612] LustreError: 68940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.466751] LustreError: 68940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2371.554139] LustreError: 71207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.608779] LustreError: 71207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2373.189349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.439713] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2373.674845] Lustre: *** cfs_fail_loc=135, val=0*** [ 2373.805263] LustreError: 72026:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2373.816158] LustreError: 72026:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.133@tcp: cfg command failed: rc = -2 [ 2373.821602] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2373.821602] [ 2373.827376] LustreError: MGC192.168.201.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2373.842560] LustreError: 72014:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2373.849799] LustreError: 72014:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 2373.872105] LustreError: 72014:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2373.891539] LustreError: 72014:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2373.914760] Lustre: server umount lustre-MDT0000 complete [ 2373.925275] LustreError: 72014:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2381.803906] LustreError: 68940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.834908] LustreError: 68940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2383.200977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.397677] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2383.413257] LustreError: 72371:0:(ldlm_resource.c:1227:ldlm_resource_complain()) MGC192.168.201.133@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f3e06473) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2383.439323] LustreError: 68934:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2383.446267] LustreError: 67997:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000028b6013 x1877971159775616/t0(0) o250->MGC192.168.201.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2383.607857] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (not set up) [ 2383.835965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.920652] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2388.726448] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2388.985550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2389.027693] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2389.071794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2389.133196] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2399.205634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2399.219138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.225447] Lustre: Skipped 1 previous similar message [ 2399.242822] Lustre: Skipped 1 previous similar message [ 2403.451461] Lustre: server umount lustre-OST0000 complete [ 2409.441235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.460985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.474590] Lustre: Skipped 1 previous similar message [ 2414.169676] Lustre: server umount lustre-MDT0000 complete [ 2414.564633] LustreError: 68940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.599442] LustreError: 68940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2418.444843] LustreError: 68922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973008 with bad export cookie 865882826399558486 [ 2418.460543] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2418.792995] Lustre: server umount lustre-MDT0001 complete [ 2429.403206] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2432.073770] Key type lgssc unregistered [ 2432.426954] LNet: 73804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2432.434119] LNetError: 73804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.453947] LNet: Removed LNI 192.168.201.133@tcp [ 2433.719827] Key type .llcrypt unregistered [ 2433.724319] Key type ._llcrypt unregistered [ 2459.885992] Key type ._llcrypt registered [ 2459.889312] Key type .llcrypt registered [ 2460.025942] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2461.195102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.278424] alg: No test for adler32 (adler32-zlib) [ 2462.457100] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2462.849952] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2464.583234] Key type lgssc registered [ 2466.621870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2475.870927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2475.907357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.661144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.311911] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2487.780066] LustreError: 75231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.815147] LustreError: 75231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2492.902562] LustreError: 75232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.401155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.997201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2502.453908] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2511.012996] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2517.485527] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2524.661440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.932583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.619066] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2538.615692] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2547.443705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2553.898994] Lustre: Failing over lustre-MDT0000 [ 2554.198592] Lustre: server umount lustre-MDT0000 complete [ 2554.639850] LustreError: 75228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.924191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2557.929201] LustreError: 76188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.945652] Lustre: Skipped 2 previous similar messages [ 2557.989407] LustreError: 76188:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2563.048438] LustreError: 75231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.065411] LustreError: 75231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2564.245787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2564.433779] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2564.548369] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2564.623222] LustreError: 78346:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2564.626810] LustreError: 78346:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.133@tcp: cfg command failed: rc = -9 [ 2564.631036] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2564.631036] [ 2564.640310] LustreError: MGC192.168.201.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2564.654816] LustreError: 78335:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2564.661232] LustreError: 78335:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -9 [ 2564.667901] LustreError: 78335:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 2564.673787] LustreError: 78335:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2564.694798] Lustre: server umount lustre-MDT0000 complete [ 2564.696845] LustreError: 78335:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2571.697397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.784503] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.977815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.004251] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2576.111837] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2576.690688] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2577.386739] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2577.420712] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2577.459670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2587.619713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2587.633855] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.636346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.784546] Lustre: server umount lustre-OST0000 complete [ 2597.857377] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2597.862607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.882724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.895819] Lustre: Skipped 2 previous similar messages [ 2597.912106] Lustre: Skipped 2 previous similar messages [ 2600.528707] Lustre: server umount lustre-MDT0000 complete [ 2602.979043] LustreError: 75231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.994619] LustreError: 75231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2604.097412] LustreError: 75213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973194 with bad export cookie 15266840271624906991 [ 2604.099145] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2604.103471] LustreError: 75213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2604.290573] Lustre: server umount lustre-MDT0001 complete [ 2614.238728] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2617.122395] Key type lgssc unregistered [ 2617.490660] LNet: 80130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2617.495513] LNetError: 80130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2617.533301] LNet: Removed LNI 192.168.201.133@tcp [ 2618.539343] Key type .llcrypt unregistered [ 2618.541673] Key type ._llcrypt unregistered [ 2633.914599] Key type ._llcrypt registered [ 2633.920656] Key type .llcrypt registered [ 2634.094330] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2634.486505] Key type .llcrypt unregistered [ 2634.489373] Key type ._llcrypt unregistered [ 2660.069368] Key type ._llcrypt registered [ 2660.071461] Key type .llcrypt registered [ 2660.171725] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2661.161668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2661.183705] alg: No test for adler32 (adler32-zlib) [ 2662.555978] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2662.892198] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2664.688796] Key type lgssc registered [ 2666.412820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2674.560697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2674.576946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.004354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.681479] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2685.422077] LustreError: 82046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.452327] LustreError: 82046:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2690.540424] LustreError: 82047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.747925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.527832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2698.227812] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2709.327383] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2717.405327] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2725.197136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2725.777636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.445525] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2742.158318] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2745.282627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2760.675115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.701539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.714186] Lustre: Skipped 1 previous similar message [ 2765.538450] Lustre: server umount lustre-OST0000 complete [ 2770.913846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.937439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2770.942380] Lustre: Skipped 2 previous similar messages [ 2776.042495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.049778] Lustre: Skipped 1 previous similar message [ 2776.250547] Lustre: server umount lustre-MDT0000 complete [ 2780.133063] LustreError: 82775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.168595] LustreError: 82775:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2780.716150] LustreError: 82027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973370 with bad export cookie 12747676259706102615 [ 2780.722957] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2780.730439] LustreError: 82027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2781.155428] Lustre: server umount lustre-MDT0001 complete [ 2793.197562] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2797.332534] Key type lgssc unregistered [ 2797.679481] LNet: 85600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2797.697577] LNetError: 85600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2797.747350] LNet: Removed LNI 192.168.201.133@tcp [ 2798.792349] Key type .llcrypt unregistered [ 2798.794346] Key type ._llcrypt unregistered [ 2827.100972] Key type ._llcrypt registered [ 2827.103386] Key type .llcrypt registered [ 2827.271158] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2828.599890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2828.693244] alg: No test for adler32 (adler32-zlib) [ 2829.799450] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2830.020370] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2831.671251] Key type lgssc registered [ 2833.046779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2841.785447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 2841.825070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.425063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.239736] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2853.857177] LustreError: 87030:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.890078] LustreError: 87030:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2858.003643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2858.543588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2863.698232] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2871.704664] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2879.653270] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2888.794279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.374768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.401479] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2907.396375] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2913.980900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2924.513245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2924.527714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2924.548152] LustreError: Skipped 1 previous similar message [ 2924.558123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2924.564714] Lustre: Skipped 1 previous similar message [ 2929.640864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2929.650747] Lustre: Skipped 2 previous similar messages [ 2930.868618] Lustre: server umount lustre-OST0000 complete [ 2939.889674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.897930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2939.919156] Lustre: Skipped 1 previous similar message [ 2941.849162] Lustre: server umount lustre-MDT0000 complete [ 2945.001537] LustreError: 87026:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.040834] LustreError: 87026:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2946.042712] LustreError: 87010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973535 with bad export cookie 17207659183981204225 [ 2946.047944] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.053872] LustreError: 87010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2946.477777] Lustre: server umount lustre-MDT0001 complete [ 2959.893265] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2962.982035] Key type lgssc unregistered [ 2963.242286] LNet: 90586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2963.247987] LNetError: 90586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2964.263794] LNet: Removed LNI 192.168.201.133@tcp [ 2965.271220] Key type .llcrypt unregistered [ 2965.273542] Key type ._llcrypt unregistered [ 2990.604798] Key type ._llcrypt registered [ 2990.608391] Key type .llcrypt registered [ 2990.724917] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2992.123873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2992.172370] alg: No test for adler32 (adler32-zlib) [ 2993.372749] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2993.699487] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2995.479231] Key type lgssc registered [ 2996.650287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3004.440530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3004.455771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.776404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.806909] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3015.152084] LustreError: 92017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3015.171984] LustreError: 92017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3020.263158] LustreError: 92016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.319221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.809134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3024.701832] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3032.638910] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3038.772422] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3046.520561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3047.003235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.141987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3052.941874] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3061.302772] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3079.512430] Lustre: server umount lustre-OST0000 complete [ 3086.317904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.330125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.426212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.429028] Lustre: Skipped 1 previous similar message [ 3093.699820] Lustre: server umount lustre-MDT0000 complete [ 3096.546787] LustreError: 92017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.565191] LustreError: 92017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3097.570786] Lustre: 91317:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790973671/real 1790973671] req@000000006255dc78 x1877971915796864/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1790973687 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3097.618740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.645968] Lustre: Skipped 1 previous similar message [ 3097.952399] LustreError: 91998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973687 with bad export cookie 11594931726056874223 [ 3097.958961] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3097.964816] LustreError: 91998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3098.312845] Lustre: server umount lustre-MDT0001 complete [ 3109.584844] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3112.493300] Key type lgssc unregistered [ 3112.792719] LNet: 95568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3112.798530] LNetError: 95568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3113.836905] LNet: Removed LNI 192.168.201.133@tcp [ 3114.695406] Key type .llcrypt unregistered [ 3114.698291] Key type ._llcrypt unregistered [ 3136.495178] Key type ._llcrypt registered [ 3136.496895] Key type .llcrypt registered [ 3136.590522] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3138.224072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3138.247350] alg: No test for adler32 (adler32-zlib) [ 3139.193231] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3139.430107] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3141.055211] Key type lgssc registered [ 3142.754486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3152.601934] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3152.637458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3168.545529] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3173.664208] LustreError: 96963:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 3200.353641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3206.252479] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3213.333940] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3217.933656] Lustre: server umount lustre-OST0000 complete [ 3225.075685] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 16:43:33 (1790973813) [ 3232.590951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3232.896666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3237.374948] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3238.377677] LustreError: 98601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3238.401707] LustreError: 98601:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3242.497613] LustreError: 98600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.863773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.213345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3250.793731] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3257.392738] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3262.193310] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3267.356052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3267.621846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.356832] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3277.890518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3280.277508] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3292.135921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3292.140152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3292.146471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.160706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.160954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3293.171690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3296.200295] Lustre: server umount lustre-OST0000 complete [ 3302.372777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3302.393640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.408665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.418267] Lustre: Skipped 1 previous similar message [ 3306.842676] Lustre: server umount lustre-MDT0000 complete [ 3308.515166] LustreError: 98595:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.555060] LustreError: 98595:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3311.526384] LustreError: 100443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790973901 with bad export cookie 8627588216814568200 [ 3311.536166] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3311.551065] LustreError: 100443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3311.997368] Lustre: server umount lustre-MDT0001 complete [ 3324.991091] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3328.232403] Key type lgssc unregistered [ 3328.590934] LNet: 102292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3328.597526] LNetError: 102292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3328.614143] LNet: Removed LNI 192.168.201.133@tcp [ 3329.605298] Key type .llcrypt unregistered [ 3329.606972] Key type ._llcrypt unregistered [ 3373.795688] Key type ._llcrypt registered [ 3373.796927] Key type .llcrypt registered [ 3374.118845] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3375.899719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3375.924249] alg: No test for adler32 (adler32-zlib) [ 3377.026800] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3377.337189] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3379.106379] Key type lgssc registered [ 3380.323606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3389.585236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3389.657167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3391.136834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.906408] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3401.703339] LustreError: 104505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.732148] LustreError: 104505:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3406.820286] LustreError: 104504:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.149615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.539485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3412.639761] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3420.212756] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3426.473421] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3433.829555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3434.324507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.408550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3440.993435] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3449.221639] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3463.142237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.152772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3463.154297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3463.155256] Lustre: Skipped 1 previous similar message [ 3468.258668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3468.406657] Lustre: server umount lustre-OST0000 complete [ 3473.377915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.399541] Lustre: Skipped 1 previous similar message [ 3473.424552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3473.426809] Lustre: Skipped 1 previous similar message [ 3478.496339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3478.501518] Lustre: Skipped 1 previous similar message [ 3478.949627] Lustre: server umount lustre-MDT0000 complete [ 3483.509720] LustreError: 104485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974073 with bad export cookie 977545315654293156 [ 3483.517734] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3483.524417] LustreError: 104485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3484.323308] Lustre: server umount lustre-MDT0001 complete [ 3495.311603] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3499.192147] Key type lgssc unregistered [ 3499.507859] LNet: 108057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3499.512136] LNetError: 108057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3499.533959] LNet: Removed LNI 192.168.201.133@tcp [ 3500.925853] Key type .llcrypt unregistered [ 3500.928110] Key type ._llcrypt unregistered [ 3521.102442] Key type ._llcrypt registered [ 3521.104389] Key type .llcrypt registered [ 3521.183186] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3522.369083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3522.604753] alg: No test for adler32 (adler32-zlib) [ 3524.116328] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3524.385731] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3526.127208] Key type lgssc registered [ 3527.930385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3536.712875] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 3536.732379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3552.353370] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3557.471820] LustreError: 109319:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 3584.159426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.020285] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3598.087419] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3604.930730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.179251] LustreError: MGC192.168.201.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3605.187706] LustreError: 110345:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3605.191827] LustreError: 110345:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 3605.196221] LustreError: 110345:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 3605.266784] Lustre: server umount lustre-MDT0000 complete [ 3605.269333] LustreError: 110345:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3625.439776] Lustre: 108645:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790974199/real 1790974199] req@000000000a25bf93 x1877972472565120/t0(0) o400->MGC192.168.201.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790974215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3625.482476] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3625.500050] LustreError: 109332:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3646.082454] Lustre: server umount lustre-OST0000 complete [ 3655.729161] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 3667.263437] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3703.706987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.152485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3704.226048] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3704.379352] Lustre: lustre-MDT0000: new disk, initializing [ 3704.503233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.519541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3709.158143] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3719.732968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.797893] Lustre: 113140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3719.867568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3719.871125] Lustre: Skipped 1 previous similar message [ 3719.939400] Lustre: lustre-MDT0001: new disk, initializing [ 3720.034284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3720.083862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3720.100278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3723.270721] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3733.989128] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3740.010852] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3747.781107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3747.985334] Lustre: lustre-OST0000: new disk, initializing [ 3747.988585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3747.994719] Lustre: 115108:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3748.054197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3754.652351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3754.682739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3754.862207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3755.608299] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3766.021468] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3771.210907] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3771.571384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3775.570162] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3775.771629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3780.579105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.588740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.598742] LustreError: Skipped 1 previous similar message [ 3780.612731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3780.620751] Lustre: Skipped 1 previous similar message [ 3783.614046] Lustre: server umount lustre-OST0000 complete [ 3790.836132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3790.850201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3790.878593] Lustre: Skipped 2 previous similar messages [ 3790.916033] Lustre: Skipped 1 previous similar message [ 3794.390868] Lustre: server umount lustre-MDT0000 complete [ 3799.382546] LustreError: 113936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974389 with bad export cookie 2357319029102498813 [ 3799.393985] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3799.416981] LustreError: 113936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3800.322590] Lustre: server umount lustre-MDT0001 complete [ 3810.656682] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 16:53:18 (1790974398) [ 3815.085767] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3835.437391] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 3845.262577] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3882.115533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.346341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3882.369737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3882.420854] Lustre: lustre-MDT0000: new disk, initializing [ 3882.476891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3882.487295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3886.096253] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3897.906364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.023909] Lustre: 120010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3898.170760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3898.174301] Lustre: Skipped 1 previous similar message [ 3898.300310] Lustre: lustre-MDT0001: new disk, initializing [ 3898.473879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3898.591480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3898.602370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3903.046207] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3913.026606] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3918.118920] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3924.330965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3924.580534] Lustre: lustre-OST0000: new disk, initializing [ 3924.585163] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3924.599651] Lustre: 121976:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3924.697275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.350251] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3931.695457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3931.704909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3931.791489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3939.089218] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3943.294950] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3943.487030] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3948.298933] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3948.571467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3952.609217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.611152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3952.625191] Lustre: Skipped 1 previous similar message [ 3956.991051] Lustre: server umount lustre-OST0000 complete [ 3962.851730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3962.867453] Lustre: Skipped 1 previous similar message [ 3962.868512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3962.900387] Lustre: Skipped 2 previous similar messages [ 3966.685590] Lustre: server umount lustre-MDT0000 complete [ 3967.969040] LustreError: 120017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3967.985286] LustreError: 120017:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3970.470327] LustreError: 120001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974560 with bad export cookie 2357319029102499695 [ 3970.490462] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3970.901672] Lustre: server umount lustre-MDT0001 complete [ 3979.454815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.049714] LustreError: 124196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.146970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.987601] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3985.378432] LustreError: 124197:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3990.500409] LustreError: 124196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.089673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.251783] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4007.311615] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4012.964054] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4020.153149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4026.638940] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4034.569672] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4044.585599] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4050.400597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4050.410918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4050.425833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4050.439046] Lustre: Skipped 1 previous similar message [ 4055.187907] Lustre: server umount lustre-OST0000 complete [ 4060.646059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.664399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4060.674304] Lustre: Skipped 2 previous similar messages [ 4065.674883] Lustre: server umount lustre-MDT0000 complete [ 4065.773025] LustreError: 124924:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4065.799883] LustreError: 124924:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4069.420550] LustreError: 124177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974659 with bad export cookie 2357319029102500577 [ 4069.425094] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4069.430279] LustreError: 124177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4069.837779] Lustre: server umount lustre-MDT0001 complete [ 4082.120848] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4084.568894] Key type lgssc unregistered [ 4084.834183] LNet: 127825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4084.837478] LNetError: 127825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4084.853662] LNet: Removed LNI 192.168.201.133@tcp [ 4085.518437] Key type .llcrypt unregistered [ 4085.525573] Key type ._llcrypt unregistered [ 4104.184183] Key type ._llcrypt registered [ 4104.189207] Key type .llcrypt registered [ 4104.318129] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 4121.806626] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4122.633537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4122.651029] alg: No test for adler32 (adler32-zlib) [ 4123.603068] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 4123.771388] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4125.375185] Key type lgssc registered [ 4126.760730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4163.044073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 4163.097327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4164.344255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4164.364375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4164.453389] Lustre: lustre-MDT0000: new disk, initializing [ 4164.535490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4164.566932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4169.724247] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4180.077727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4180.284637] Lustre: 130994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4180.360273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4180.368854] Lustre: Skipped 1 previous similar message [ 4180.518969] Lustre: lustre-MDT0001: new disk, initializing [ 4180.666751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4180.745653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4180.779556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4184.471950] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4194.595797] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4200.069593] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4206.081921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4206.464851] Lustre: lustre-OST0000: new disk, initializing [ 4206.469064] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4206.474300] Lustre: 132962:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4206.551684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4212.375538] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4212.799357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4212.804191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4212.879802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4222.384527] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4228.598922] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4228.832126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4234.900826] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4235.210310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4238.819174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.853300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4238.861252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4238.878470] Lustre: Skipped 1 previous similar message [ 4243.822636] Lustre: server umount lustre-OST0000 complete [ 4249.062419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4249.072932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4249.096614] Lustre: Skipped 2 previous similar messages [ 4254.184537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4254.197580] Lustre: Skipped 2 previous similar messages [ 4258.277816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4258.283925] Lustre: Skipped 1 previous similar message [ 4262.367435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4262.687488] Lustre: server umount lustre-MDT0000 complete [ 4266.353853] LustreError: 130984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974856 with bad export cookie 17895679734281060565 [ 4266.358475] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4266.363450] LustreError: 130984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4266.653366] Lustre: server umount lustre-MDT0001 complete [ 4275.916474] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 17:01:04 (1790974864) [ 4283.226258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4283.819116] LustreError: 135389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.836502] LustreError: 135389:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4283.941270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.920398] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4289.013650] LustreError: 135390:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4293.089871] LustreError: 135389:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4297.518400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.957623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4301.625738] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4307.250306] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4312.539268] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4318.701487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4318.713909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.741778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4321.166823] Lustre: server umount lustre-MDT0000 complete [ 4322.849142] LustreError: 136903:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4322.875147] LustreError: 136903:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4325.000813] LustreError: 135371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790974914 with bad export cookie 17895679734281061447 [ 4325.009360] LustreError: 135371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4325.009745] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4325.427229] Lustre: server umount lustre-MDT0001 complete [ 4333.104529] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 17:02:01 (1790974921) [ 4340.145783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4355.743631] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4360.865988] LustreError: 137803:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 4386.592401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.827532] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4400.449793] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4402.737612] Lustre: server umount lustre-OST0000 complete [ 4411.321168] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 17:03:19 (1790974999) [ 4418.598378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4419.281952] LustreError: 139248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4419.440220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.398354] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4429.795803] LustreError: 139249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4429.822574] LustreError: 139249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4431.725951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.605511] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4443.615571] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4448.554678] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4456.104907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4456.721142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4456.727440] Lustre: Skipped 1 previous similar message [ 4465.392678] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4474.454402] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4483.554478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4483.560497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.576212] Lustre: Skipped 1 previous similar message [ 4483.583546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4483.590616] Lustre: Skipped 1 previous similar message [ 4489.741061] Lustre: server umount lustre-MDT0000 complete [ 4490.725197] LustreError: 139248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.757392] LustreError: 139248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4493.795516] LustreError: 139983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975083 with bad export cookie 17895679734281062112 [ 4493.796843] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4493.803143] LustreError: 139983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4494.353879] Lustre: server umount lustre-MDT0001 complete [ 4508.713189] Lustre: server umount lustre-OST0000 complete [ 4516.878996] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 17:05:05 (1790975105) [ 4524.041121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.729603] LustreError: 142902:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4524.829295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4529.821245] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4539.633124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4547.392682] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4555.198057] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4560.992774] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4567.680132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4569.148766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4573.502393] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4581.153307] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4585.753946] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4586.058537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4590.426813] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4590.610201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4594.659304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4594.683555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4594.721660] Lustre: Skipped 2 previous similar messages [ 4594.725037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4594.727480] Lustre: Skipped 3 previous similar messages [ 4598.693604] Lustre: server umount lustre-OST0000 complete [ 4604.900151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.919089] Lustre: Skipped 2 previous similar messages [ 4608.998038] Lustre: server umount lustre-MDT0000 complete [ 4610.025258] LustreError: 143630:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.054013] LustreError: 143630:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4612.679412] LustreError: 142883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975202 with bad export cookie 17895679734281063183 [ 4612.683268] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4612.691692] LustreError: 142883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4613.161529] Lustre: server umount lustre-MDT0001 complete [ 4621.470177] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 17:06:49 (1790975209) [ 4628.035613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4643.679727] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4648.799496] LustreError: 147060:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 4674.463596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4674.473572] Lustre: Skipped 2 previous similar messages [ 4681.192604] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4688.574980] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4695.492981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4696.146156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4700.973248] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4709.507843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4714.680683] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4722.234918] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4728.426224] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4733.234646] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4733.496685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4738.547639] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4738.768509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4745.699845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4745.701204] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4745.705900] LustreError: Skipped 1 previous similar message [ 4745.707890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4745.716219] Lustre: Skipped 1 previous similar message [ 4745.728478] Lustre: Skipped 4 previous similar messages [ 4747.020551] Lustre: server umount lustre-MDT0000 complete [ 4750.684030] LustreError: 148839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975340 with bad export cookie 17895679734281064205 [ 4750.689444] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4750.694815] LustreError: 148839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4750.820181] LustreError: 148096:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4750.834384] LustreError: 148096:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4750.976356] Lustre: server umount lustre-MDT0001 complete [ 4760.459949] Lustre: server umount lustre-OST0000 complete [ 4770.065768] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 17:09:17 (1790975357) [ 4777.300721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.055604] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4798.175469] LustreError: 151252:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 4831.728190] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4839.432719] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4846.266210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.774302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4851.852266] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4860.602901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.593148] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4872.324253] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4877.694551] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4884.668357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4884.824897] Lustre: lustre-OST0001: new disk, initializing [ 4884.828519] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4884.835300] Lustre: 154125:0:(osd_compat.c:1354:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4890.852959] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4892.767376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4892.785227] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4892.938215] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4900.267869] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4905.967367] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4906.195147] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4911.418384] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4911.634901] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4917.216289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4917.237224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4917.252027] Lustre: Skipped 2 previous similar messages [ 4917.265845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4917.277474] Lustre: Skipped 1 previous similar message [ 4920.050804] Lustre: server umount lustre-OST0000 complete [ 4930.460931] Lustre: server umount lustre-OST0001 complete [ 4940.463931] Lustre: server umount lustre-MDT0000 complete [ 4944.380782] LustreError: 155872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975534 with bad export cookie 17895679734281065157 [ 4944.396623] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4944.399117] LustreError: 155872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4944.917144] Lustre: server umount lustre-MDT0001 complete [ 4971.539198] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 17:12:39 (1790975559) [ 4973.612241] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4975.409143] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4976.933169] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 17:12:45 (1790975565) [ 4983.886160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4984.119379] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4984.152812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4984.481275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4984.487206] Lustre: Skipped 6 previous similar messages [ 4988.133026] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4996.280236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.441028] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4996.518753] Lustre: 157688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5001.260234] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5010.016185] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5017.336928] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5023.603725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5023.802364] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5029.685896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5033.811368] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5043.699304] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5049.865796] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5050.120971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5055.773435] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5056.021318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5060.578924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5060.580896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5060.586601] Lustre: Skipped 5 previous similar messages [ 5060.599730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5060.602492] LustreError: Skipped 3 previous similar messages [ 5060.641029] Lustre: Skipped 8 previous similar messages [ 5064.599667] Lustre: server umount lustre-OST0000 complete [ 5089.147665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5097.379902] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5100.929438] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.947530] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5100.956223] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5100.974427] Lustre: Skipped 1 previous similar message [ 5106.678969] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5112.895182] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5113.150597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5118.248891] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5118.569727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5124.968397] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff966708c51000.ost_server_uuid 50 [ 5126.753438] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff966708c51000.ost_server_uuid in FULL state after 0 sec [ 5126.941950] LustreError: 157695:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.33@tcp) returned error from blocking AST (req@000000003d8793a9 x1877973100926208 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000ac390062/0xf85a39e727fadc1f lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.33@tcp remote: 0xd801f8eba57dd610 expref: 5 pid: 157695 timeout: 5226 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5126.976357] LustreError: lustre-MDT0000: A client on nid 192.168.201.33@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5126.989323] LustreError: 157684:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.33@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000ac390062/0xf85a39e727fadc1f lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.33@tcp remote: 0xd801f8eba57dd610 expref: 6 pid: 157695 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5132.784376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5138.183229] Lustre: server umount lustre-OST0000 complete [ 5148.610992] Lustre: server umount lustre-MDT0000 complete [ 5152.224269] LustreError: 157700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.249817] LustreError: 157700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 5152.603635] LustreError: 157680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975742 with bad export cookie 17895679734281066340 [ 5152.608799] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5152.613820] LustreError: 157680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5159.274285] Lustre: server umount lustre-MDT0001 complete [ 5171.657193] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5174.591329] Key type lgssc unregistered [ 5174.805933] LNet: 163802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5174.821257] LNetError: 163802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5174.849354] LNet: Removed LNI 192.168.201.133@tcp [ 5175.688676] Key type .llcrypt unregistered [ 5175.691278] Key type ._llcrypt unregistered [ 5202.258744] Key type ._llcrypt registered [ 5202.261250] Key type .llcrypt registered [ 5202.377569] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5203.540419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5203.578518] alg: No test for adler32 (adler32-zlib) [ 5204.582632] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 5204.806276] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 5206.487183] Key type lgssc registered [ 5207.923449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5216.589308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5216.623246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.263351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5223.113192] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5228.518703] LustreError: 165233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5228.534686] LustreError: 165233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5231.656246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5232.132952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5236.767611] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5244.513601] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5250.472310] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5257.745946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5258.184295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5263.354804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5264.113831] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5272.397688] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5282.604681] Lustre: Failing over lustre-MDT0000 [ 5282.912531] Lustre: server umount lustre-MDT0000 complete [ 5283.300234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5283.303102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.338051] LustreError: 165233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5286.887509] LustreError: 165234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5286.894282] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.908955] LustreError: 165234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5286.927897] Lustre: Skipped 1 previous similar message [ 5292.011244] LustreError: 165962:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5292.032077] LustreError: 165962:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5295.105355] LustreError: 165215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.33@tcp arrived at 1790975885 with bad export cookie 10252159137450979856 [ 5295.115786] LustreError: 165215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5297.121813] LustreError: 165233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5297.147838] LustreError: 165233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5302.240145] Lustre: 164535:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790975876/real 1790975876] req@0000000037f19c8d x1877974234197248/t0(0) o400->MGC192.168.201.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790975892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5302.283133] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5302.944696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5307.360220] LustreError: 165233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5307.394544] LustreError: 165233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5313.511788] LustreError: 164532:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000048287637 x1877974234202496/t0(0) o250->MGC192.168.201.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5313.819448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5313.862044] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5318.873635] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5319.160299] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5319.172956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5320.867422] Lustre: lustre-MDT0000: Denying connection for new client c2a6aabe-bf4b-49c2-aae1-305f2b0889af (at 192.168.201.33@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5329.378520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5329.398575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5329.415527] Lustre: Skipped 1 previous similar message [ 5334.496755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5334.507031] Lustre: Skipped 1 previous similar message [ 5339.628819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5339.640206] Lustre: Skipped 1 previous similar message [ 5343.199142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5343.569669] Lustre: server umount lustre-OST0000 complete [ 5347.301735] LustreError: 168965:0:(ldlm_lib.c:3038:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5347.308023] Lustre: 168313:0:(ldlm_lib.c:2438:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5347.318834] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5347.345719] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5347.354874] LustreError: 168313:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000015688bc x1877974234218624/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5347.364977] LustreError: 168313:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5347.370846] LustreError: 168313:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5347.377833] Lustre: lustre-MDT0000: Recovery over after 0:28, of 2 clients 0 recovered and 2 were evicted. [ 5347.685739] Lustre: server umount lustre-MDT0000 complete [ 5349.862109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.867991] Lustre: Skipped 1 previous similar message [ 5349.870846] LustreError: 168292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5349.890937] LustreError: 168292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 5351.288842] LustreError: 164532:0:(import.c:752:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5351.290475] LustreError: 165214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790975941 with bad export cookie 10252159137450979961 [ 5351.294305] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5351.306890] LustreError: 165214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5353.664574] Lustre: server umount lustre-MDT0001 complete [ 5364.349914] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5367.025396] Key type lgssc unregistered [ 5367.256530] LNet: 169715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5367.259557] LNetError: 169715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5367.271321] LNet: Removed LNI 192.168.201.133@tcp [ 5368.046063] Key type .llcrypt unregistered [ 5368.048113] Key type ._llcrypt unregistered [ 5392.826662] Key type ._llcrypt registered [ 5392.840763] Key type .llcrypt registered [ 5393.060154] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5394.710528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5394.771861] alg: No test for adler32 (adler32-zlib) [ 5396.426707] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 5396.931378] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 5398.727400] Key type lgssc registered [ 5400.111840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5408.584787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5408.605635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.025873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5414.084306] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5419.508265] LustreError: 171144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5419.520809] LustreError: 171144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5422.046620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5422.520895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5427.744748] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5435.234296] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5440.620557] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5446.708317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5447.022639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.559977] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5454.197043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5461.641264] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5468.641429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5468.659106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5468.682935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5469.664309] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5469.673925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5469.680284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5469.711581] Lustre: Skipped 1 previous similar message [ 5470.543554] Lustre: server umount lustre-OST0000 complete [ 5474.785097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5474.795023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5474.814176] Lustre: Skipped 1 previous similar message [ 5479.909065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5479.911697] Lustre: Skipped 1 previous similar message [ 5480.764735] Lustre: server umount lustre-MDT0000 complete [ 5484.764973] LustreError: 171123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976074 with bad export cookie 7531049964542660113 [ 5484.779493] LustreError: 171123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5484.780536] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5485.310983] Lustre: server umount lustre-MDT0001 complete [ 5499.191236] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5503.164935] Key type lgssc unregistered [ 5503.657920] LNet: 174675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5503.670435] LNetError: 174675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5504.741833] LNet: Removed LNI 192.168.201.133@tcp [ 5505.938245] Key type .llcrypt unregistered [ 5505.939472] Key type ._llcrypt unregistered [ 5544.435257] Key type ._llcrypt registered [ 5544.436897] Key type .llcrypt registered [ 5544.596938] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5546.025972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5546.310547] alg: No test for adler32 (adler32-zlib) [ 5547.599724] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 5548.012681] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 5549.719516] Key type lgssc registered [ 5551.196658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5559.223680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5559.248349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5560.760863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5565.686605] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5571.051067] LustreError: 176395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.072951] LustreError: 176395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5575.518244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5575.983022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5580.504275] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5587.465484] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5592.971298] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5598.423482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5598.724972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5604.835050] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5605.891203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5613.167537] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5621.219393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5621.243467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5621.244542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5621.269692] Lustre: Skipped 1 previous similar message [ 5622.166448] Lustre: server umount lustre-OST0000 complete [ 5626.336818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.350062] Lustre: Skipped 1 previous similar message [ 5626.364199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5626.374194] Lustre: Skipped 1 previous similar message [ 5631.459173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5631.467208] Lustre: Skipped 1 previous similar message [ 5632.281148] Lustre: server umount lustre-MDT0000 complete [ 5635.896725] LustreError: 176377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976225 with bad export cookie 6237812699434203310 [ 5635.905306] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5635.917535] LustreError: 176377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5636.314217] Lustre: server umount lustre-MDT0001 complete [ 5648.764561] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5652.617639] Key type lgssc unregistered [ 5652.947661] LNet: 179926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5652.953553] LNetError: 179926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5652.983367] LNet: Removed LNI 192.168.201.133@tcp [ 5654.075654] Key type .llcrypt unregistered [ 5654.077906] Key type ._llcrypt unregistered [ 5679.153516] Key type ._llcrypt registered [ 5679.156990] Key type .llcrypt registered [ 5679.284236] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5680.355433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5680.405061] alg: No test for adler32 (adler32-zlib) [ 5681.574487] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 5681.799200] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 5683.447212] Key type lgssc registered [ 5685.107404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5695.552450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5695.590442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.005099] Lustre: *** cfs_fail_loc=135, val=0*** [ 5697.110117] LustreError: 181468:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5697.128553] LustreError: 181468:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.133@tcp: cfg command failed: rc = -2 [ 5697.141732] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5697.141732] [ 5697.152858] LustreError: MGC192.168.201.133@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5697.163054] LustreError: 181430:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5697.168502] LustreError: 181430:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 5697.179600] LustreError: 181430:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 5697.185677] LustreError: 181430:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5697.252822] Lustre: server umount lustre-MDT0000 complete [ 5697.254988] LustreError: 181430:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5703.362274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.945565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5708.884976] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5713.377908] LustreError: 181840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.397435] LustreError: 181840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5716.691697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.995235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5721.646714] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5728.968324] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5734.357594] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5737.440426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5737.448782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.481431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5738.473304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5738.473990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5738.501083] Lustre: Skipped 1 previous similar message [ 5741.601977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5741.612348] Lustre: Skipped 1 previous similar message [ 5742.530434] Lustre: server umount lustre-MDT0000 complete [ 5746.253900] LustreError: 181820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976336 with bad export cookie 6019898425365966031 [ 5746.271266] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5746.564236] Lustre: server umount lustre-MDT0001 complete [ 5757.885345] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5760.688827] Key type lgssc unregistered [ 5761.019683] LNet: 184157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5761.035067] LNetError: 184157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5761.057504] LNet: Removed LNI 192.168.201.133@tcp [ 5761.988583] Key type .llcrypt unregistered [ 5761.990834] Key type ._llcrypt unregistered [ 5782.700452] Key type ._llcrypt registered [ 5782.703160] Key type .llcrypt registered [ 5782.780343] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5783.035541] Key type .llcrypt unregistered [ 5783.037796] Key type ._llcrypt unregistered [ 5800.692808] Key type ._llcrypt registered [ 5800.694239] Key type .llcrypt registered [ 5800.798968] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5801.785421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5801.805054] alg: No test for adler32 (adler32-zlib) [ 5802.929226] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 5803.259881] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 5805.031220] Key type lgssc registered [ 5806.402889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5813.792620] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 5813.810211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5829.472435] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5834.591340] LustreError: 186311:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 5861.343375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5869.637837] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5877.925180] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5885.125273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5885.474990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5885.562558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5889.680983] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5890.536490] LustreError: 186337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5890.566899] LustreError: 186337:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5895.659630] LustreError: 187348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.704844] LustreError: 187348:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5900.153289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.809050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5907.184605] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5915.086573] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5920.956180] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5923.939204] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5936.613646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5936.625755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.636834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.642710] Lustre: Skipped 1 previous similar message [ 5941.732188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5941.745880] Lustre: Skipped 1 previous similar message [ 5946.853687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5946.870263] Lustre: Skipped 1 previous similar message [ 5947.871355] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5948.041486] Lustre: server umount lustre-OST0000 complete [ 5951.971183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.979566] Lustre: Skipped 1 previous similar message [ 5951.982508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5951.984790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.997432] LustreError: Skipped 1 previous similar message [ 5952.009230] Lustre: Skipped 1 previous similar message [ 5957.093650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5957.106517] Lustre: Skipped 1 previous similar message [ 5965.279155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5965.747942] Lustre: server umount lustre-MDT0000 complete [ 5967.342280] LustreError: 187353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5967.353595] LustreError: 187353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5969.990373] LustreError: 186317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976559 with bad export cookie 7570018312926373119 [ 5969.996594] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5970.001142] LustreError: 186317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5970.821326] Lustre: server umount lustre-MDT0001 complete [ 5981.430429] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5983.617452] Key type lgssc unregistered [ 5983.863831] LNet: 190279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5983.869352] LNetError: 190279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5983.883587] LNet: Removed LNI 192.168.201.133@tcp [ 5984.550170] Key type .llcrypt unregistered [ 5984.551754] Key type ._llcrypt unregistered [ 6006.790926] Key type ._llcrypt registered [ 6006.794330] Key type .llcrypt registered [ 6006.915779] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6008.283263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6008.687841] alg: No test for adler32 (adler32-zlib) [ 6009.816819] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 6010.011959] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 6011.663224] Key type lgssc registered [ 6013.010333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6022.498822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6022.511893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6024.122153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.706708] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6033.382771] LustreError: 191709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6033.410399] LustreError: 191709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6038.505793] LustreError: 191710:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6038.513681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6038.766369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6043.878716] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6050.066922] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6054.735900] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6062.231845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.712516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6068.849137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6069.927835] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6077.794606] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6082.646497] Lustre: Failing over lustre-MDT0000 [ 6082.845459] Lustre: server umount lustre-MDT0000 complete [ 6084.368981] LustreError: 191706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6084.582541] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.621663] Lustre: Skipped 2 previous similar messages [ 6089.466479] LustreError: 191706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6089.507553] LustreError: 191706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6094.596637] LustreError: 191704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.633169] LustreError: 191704:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6100.896072] Lustre: 191010:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790976674/real 1790976674] req@0000000026590751 x1877975078298368/t0(0) o400->MGC192.168.201.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1790976690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6100.957044] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6103.688145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6103.785638] LustreError: 191705:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.795941] LustreError: 191705:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6110.664651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6110.758696] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6110.962571] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6115.805299] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6115.841978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6115.874864] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6115.933038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6120.149949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6129.239514] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6146.016323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6146.032360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6146.059719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6150.700504] Lustre: server umount lustre-OST0000 complete [ 6156.772253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6156.775014] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6156.778876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6156.778887] Lustre: Skipped 1 previous similar message [ 6156.803546] Lustre: Skipped 2 previous similar messages [ 6160.961548] Lustre: server umount lustre-MDT0000 complete [ 6161.888791] LustreError: 191704:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6161.913023] LustreError: 191704:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 6165.225961] LustreError: 191689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976755 with bad export cookie 7562335480710106362 [ 6165.234157] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6165.240054] LustreError: 191689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6165.970308] Lustre: server umount lustre-MDT0001 complete [ 6178.409314] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 6181.953317] Key type lgssc unregistered [ 6182.366973] LNet: 196670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6182.377915] LNetError: 196670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6183.403067] LNet: Removed LNI 192.168.201.133@tcp [ 6184.706842] Key type .llcrypt unregistered [ 6184.715223] Key type ._llcrypt unregistered [ 6212.310713] Key type ._llcrypt registered [ 6212.313947] Key type .llcrypt registered [ 6212.467665] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6213.780923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6214.009205] alg: No test for adler32 (adler32-zlib) [ 6215.130947] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 6215.360105] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 6217.047202] Key type lgssc registered [ 6218.128853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6226.792742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6226.824531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6228.200908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.790687] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6238.697480] LustreError: 198099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6238.720697] LustreError: 198099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6243.810395] LustreError: 198100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.633536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.179584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6250.636769] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6259.653273] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6266.173513] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6272.885045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6273.458816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6274.499255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6281.236596] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6289.448446] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6293.830405] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6299.790384] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6319.538201] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6330.850126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6330.855909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6330.872033] LustreError: Skipped 1 previous similar message [ 6330.873200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6330.904164] Lustre: Skipped 1 previous similar message [ 6335.909973] Lustre: server umount lustre-OST0000 complete [ 6341.094937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6341.104559] Lustre: Skipped 1 previous similar message [ 6341.106558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6341.118666] Lustre: Skipped 2 previous similar messages [ 6346.211896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6346.219798] Lustre: Skipped 1 previous similar message [ 6346.600353] Lustre: server umount lustre-MDT0000 complete [ 6350.291911] LustreError: 198078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790976940 with bad export cookie 12584306652738204762 [ 6350.299894] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6350.978686] Lustre: server umount lustre-MDT0001 complete [ 6362.671526] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 6365.763489] Key type lgssc unregistered [ 6366.239659] LNet: 201971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6366.251875] LNetError: 201971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6366.284909] LNet: Removed LNI 192.168.201.133@tcp [ 6367.419152] Key type .llcrypt unregistered [ 6367.420964] Key type ._llcrypt unregistered [ 6390.551338] Key type ._llcrypt registered [ 6390.553707] Key type .llcrypt registered [ 6390.707460] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6392.180905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6392.321611] alg: No test for adler32 (adler32-zlib) [ 6393.543743] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 6394.000874] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 6395.785356] Key type lgssc registered [ 6397.441720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6407.396468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6407.442284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6409.247839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6415.286191] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6419.426928] LustreError: 203253:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6419.445804] LustreError: 203253:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6420.451599] LustreError: 203252:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.572480] LustreError: 203253:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.677842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6426.145309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6431.098891] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6439.306353] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6445.822453] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6453.381932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6453.847550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6455.940586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6462.120697] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6470.509229] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6476.233925] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6490.073810] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6499.722982] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6504.219527] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6516.704606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6516.717918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.733632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6516.749592] Lustre: Skipped 1 previous similar message [ 6516.774446] Lustre: Skipped 1 previous similar message [ 6521.166681] Lustre: server umount lustre-OST0000 complete [ 6526.948933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.960100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.970135] Lustre: Skipped 1 previous similar message [ 6531.717873] Lustre: server umount lustre-MDT0000 complete [ 6532.064285] LustreError: 203248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6532.081340] LustreError: 203248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6535.842022] LustreError: 205931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977125 with bad export cookie 5716897886251711316 [ 6535.843059] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6535.849609] LustreError: 205931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6536.216568] Lustre: server umount lustre-MDT0001 complete [ 6546.918184] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 6550.100794] Key type lgssc unregistered [ 6550.380292] LNet: 208177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6550.386315] LNetError: 208177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6550.411763] LNet: Removed LNI 192.168.201.133@tcp [ 6551.439522] Key type .llcrypt unregistered [ 6551.446192] Key type ._llcrypt unregistered [ 6575.767881] Key type ._llcrypt registered [ 6575.770878] Key type .llcrypt registered [ 6575.978958] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6577.448116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6577.548570] alg: No test for adler32 (adler32-zlib) [ 6578.876840] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 6579.296318] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 6581.055372] Key type lgssc registered [ 6582.740824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6592.080795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6592.100849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.721589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.103144] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6603.240166] LustreError: 209622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6603.260676] LustreError: 209622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6608.354584] LustreError: 209621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6608.916795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.412129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6615.293867] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6624.419129] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6631.684718] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6638.421512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6638.924896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.934706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6645.448957] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6653.177554] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6669.798032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.810324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6669.810482] Lustre: Skipped 1 previous similar message [ 6674.564385] Lustre: server umount lustre-OST0000 complete [ 6680.034717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.054737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6680.067176] Lustre: Skipped 2 previous similar messages [ 6684.935512] Lustre: server umount lustre-MDT0000 complete [ 6685.157950] LustreError: 209621:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.181517] LustreError: 209621:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6688.503891] LustreError: 209600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977278 with bad export cookie 15457896804378622059 [ 6688.511285] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.527208] LustreError: 209600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6689.046068] Lustre: server umount lustre-MDT0001 complete [ 6701.637795] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 6705.792808] Key type lgssc unregistered [ 6706.132718] LNet: 213304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6706.137482] LNetError: 213304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6706.161199] LNet: Removed LNI 192.168.201.133@tcp [ 6707.512792] Key type .llcrypt unregistered [ 6707.517853] Key type ._llcrypt unregistered [ 6730.811493] Key type ._llcrypt registered [ 6730.815110] Key type .llcrypt registered [ 6730.980567] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6732.462221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6732.718208] alg: No test for adler32 (adler32-zlib) [ 6733.788298] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 6734.145591] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 6735.887305] Key type lgssc registered [ 6737.449572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6745.823639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6745.872763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6747.702556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6752.778743] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6757.868526] LustreError: 214586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6757.894799] LustreError: 214586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6762.677194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6762.976970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6763.166826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6768.415191] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6776.807410] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6783.548231] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6791.119811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6791.537663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6792.603261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6798.986832] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6808.060534] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6828.525263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6828.538054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.540745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6828.566892] Lustre: Skipped 1 previous similar message [ 6828.585169] Lustre: Skipped 1 previous similar message [ 6830.425381] Lustre: server umount lustre-OST0000 complete [ 6838.754166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6838.756394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6838.767178] Lustre: Skipped 1 previous similar message [ 6838.789595] Lustre: Skipped 1 previous similar message [ 6841.572509] Lustre: server umount lustre-MDT0000 complete [ 6843.881639] LustreError: 214583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6843.923963] LustreError: 214583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6845.948935] LustreError: 214566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977435 with bad export cookie 14072619704915370252 [ 6845.961347] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6845.973473] LustreError: 214566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6846.404729] Lustre: server umount lustre-MDT0001 complete [ 6859.749768] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 6863.017481] Key type lgssc unregistered [ 6863.363772] LNet: 218270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6863.372376] LNetError: 218270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6864.423794] LNet: Removed LNI 192.168.201.133@tcp [ 6865.481683] Key type .llcrypt unregistered [ 6865.484892] Key type ._llcrypt unregistered [ 6884.856420] Key type ._llcrypt registered [ 6884.860162] Key type .llcrypt registered [ 6884.988261] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6886.742042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6887.244928] alg: No test for adler32 (adler32-zlib) [ 6888.648584] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 6889.124021] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 6890.928768] Key type lgssc registered [ 6892.915455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6903.261525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 6903.287455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6905.245909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6910.474167] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6915.573236] LustreError: 219553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6915.602376] LustreError: 219553:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6920.432223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.720391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6923.906379] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6932.269921] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6937.958485] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6947.140917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.649479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6950.715966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6955.473413] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6964.286314] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6973.806534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6973.950926] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6974.278368] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6980.644399] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6988.918546] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6990.837759] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7003.394695] Lustre: Permanently deactivating lustre-OST0001 [ 7003.451339] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7012.430488] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7024.959064] Lustre: Permanently reactivating lustre-OST0001 [ 7025.014478] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7025.022163] Lustre: Skipped 2 previous similar messages [ 7034.470961] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7034.494713] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7034.506301] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7034.521419] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7034.574865] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7043.053484] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.082546] Lustre: Skipped 1 previous similar message [ 7043.102892] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7046.585232] Lustre: server umount lustre-OST0001 complete [ 7048.166644] LustreError: 221376:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.188908] LustreError: 221376:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7053.280261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7053.288397] LustreError: 221378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7053.297544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7053.297555] Lustre: Skipped 1 previous similar message [ 7053.298516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7053.298521] Lustre: Skipped 1 previous similar message [ 7053.373169] LustreError: Skipped 1 previous similar message [ 7057.164651] Lustre: server umount lustre-OST0000 complete [ 7063.520318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7063.535069] Lustre: Skipped 1 previous similar message [ 7063.543825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7063.551505] Lustre: Skipped 2 previous similar messages [ 7067.600434] Lustre: server umount lustre-MDT0000 complete [ 7068.653055] LustreError: 221122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.683747] LustreError: 221122:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7071.750895] LustreError: 219535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977661 with bad export cookie 12427558942676744380 [ 7071.767274] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7072.235861] Lustre: server umount lustre-MDT0001 complete [ 7084.503692] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 7089.044601] Key type lgssc unregistered [ 7089.521220] LNet: 224672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7089.533377] LNetError: 224672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7089.571431] LNet: Removed LNI 192.168.201.133@tcp [ 7090.832635] Key type .llcrypt unregistered [ 7090.836537] Key type ._llcrypt unregistered [ 7133.797228] Key type ._llcrypt registered [ 7133.798896] Key type .llcrypt registered [ 7133.884896] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7134.948177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7134.990895] alg: No test for adler32 (adler32-zlib) [ 7136.142362] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 7136.480131] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 7138.215367] Key type lgssc registered [ 7139.720453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7148.395376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7148.429250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.594415] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7149.607515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7149.714197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7154.248851] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7165.475585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7165.731721] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7165.831747] Lustre: 226892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7166.516685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7174.221816] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7183.971899] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7190.803977] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7200.067929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7200.230191] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7200.486342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7208.260973] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7208.670242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7216.714319] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7219.719736] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7226.492243] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7235.496168] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7244.539985] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7260.148394] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7260.155742] Lustre: Skipped 1 previous similar message [ 7282.190925] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7282.199543] Lustre: Skipped 2 previous similar messages [ 7315.515717] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7315.526995] Lustre: Skipped 3 previous similar messages [ 7385.037367] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7385.044673] Lustre: Skipped 8 previous similar messages [ 7392.751765] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7392.771662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7392.795067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7393.255294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7397.201216] Lustre: server umount lustre-OST0000 complete [ 7401.952648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7401.963309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7401.982083] Lustre: Skipped 1 previous similar message [ 7401.986563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7401.990754] Lustre: Skipped 2 previous similar messages [ 7403.491678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7403.498470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7406.656770] Lustre: server umount lustre-MDT0000 complete [ 7409.978829] LustreError: 229464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790977999 with bad export cookie 2394216902395730808 [ 7409.979521] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7410.001958] LustreError: 229464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7410.292373] Lustre: server umount lustre-MDT0001 complete [ 7420.155543] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 7423.634659] Key type lgssc unregistered [ 7423.933558] LNet: 231692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7423.937942] LNetError: 231692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7424.998774] LNet: Removed LNI 192.168.201.133@tcp [ 7426.816906] Key type .llcrypt unregistered [ 7426.823830] Key type ._llcrypt unregistered [ 7460.444352] Key type ._llcrypt registered [ 7460.446155] Key type .llcrypt registered [ 7460.595727] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7462.033053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7462.045664] alg: No test for adler32 (adler32-zlib) [ 7463.122893] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 7463.425782] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 7465.095253] Key type lgssc registered [ 7466.140922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7475.358353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' or install '{/usr}/sbin/mount.lustre_tgt' [ 7475.394295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.234750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.346367] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7487.462863] LustreError: 233120:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7487.498274] LustreError: 233120:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7492.579453] LustreError: 233121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7494.106622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7494.604062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7499.436659] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7507.411252] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7513.202603] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7520.061925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7520.357112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7522.418507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7526.932995] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7533.916091] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7550.951094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7550.961078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.972900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7553.506570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7553.511534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7553.568153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7555.561996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7557.122102] Lustre: server umount lustre-OST0000 complete [ 7560.672808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7560.678426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7560.696581] Lustre: Skipped 1 previous similar message [ 7560.730148] Lustre: Skipped 2 previous similar messages [ 7565.803777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7565.806866] Lustre: Skipped 1 previous similar message [ 7566.967643] Lustre: server umount lustre-MDT0000 complete [ 7570.914230] LustreError: 235236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7570.937398] LustreError: 235236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7571.153086] LustreError: 235700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790978161 with bad export cookie 15791803886433976766 [ 7571.167193] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7571.168325] LustreError: 235700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7571.497372] Lustre: server umount lustre-MDT0001 complete [ 7581.878621] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 7585.227401] Key type lgssc unregistered [ 7585.496834] LNet: 236808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7585.503906] LNetError: 236808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7585.521422] LNet: Removed LNI 192.168.201.133@tcp [ 7586.814631] Key type .llcrypt unregistered [ 7586.823746] Key type ._llcrypt unregistered [ 7617.857859] Key type ._llcrypt registered [ 7617.860243] Key type .llcrypt registered [ 7618.056670] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7619.462190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7619.481286] alg: No test for adler32 (adler32-zlib) [ 7620.451859] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 7620.622347] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 7622.216930] Key type lgssc registered [ 7624.164355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7896.284351] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 7900.704829] Key type lgssc unregistered [ 7900.949584] LNet: 238856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7900.960150] LNetError: 238856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7900.978147] LNet: Removed LNI 192.168.201.133@tcp [ 7901.818524] Key type .llcrypt unregistered [ 7901.820527] Key type ._llcrypt unregistered [ 7933.525459] Key type ._llcrypt registered [ 7933.527658] Key type .llcrypt registered [ 7933.637455] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 7953.730467] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7955.389585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7955.618111] alg: No test for adler32 (adler32-zlib) [ 7957.133858] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 7957.618086] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 7959.352497] Key type lgssc registered [ 7961.123507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7999.048748] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7727 sec ========== 18:03:06 (1790978586) [ 8001.348875] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:03:08 (1790978588) === [ 8005.490555] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:03:13 (1790978593) === [ 8056.589586] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8059.814293] Key type lgssc unregistered [ 8060.132643] LNet: 244111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8060.138967] LNetError: 244111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8060.167186] LNet: Removed LNI 192.168.201.133@tcp [ 8061.285302] Key type .llcrypt unregistered [ 8061.286814] Key type ._llcrypt unregistered