[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481725329 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003322] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007993] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013079] *** VALIDATE selinux *** [ 0.021490] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026051] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028079] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] *** VALIDATE tmpfs *** [ 0.031491] *** VALIDATE proc *** [ 0.033266] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043191] debug: unmapping init [mem 0xffffffff91259000-0xffffffff91260fff] [ 0.045993] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046718] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051016] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054324] rcu: Hierarchical SRCU implementation. [ 0.056388] smp: Bringing up secondary CPUs ... [ 0.057601] x86: Booting SMP configuration: [ 0.058031] .... node #0, CPUs: #1 #2 #3 [ 0.061646] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.134525] node 0 deferred pages initialised in 67ms [ 0.137079] devtmpfs: initialized [ 0.138138] x86/mm: Memory block size: 128MB [ 0.140405] gcov: version magic: 0x41383552 [ 0.142277] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145101] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147399] pinctrl core: initialized pinctrl subsystem [ 0.149184] [ 0.149804] ************************************************************* [ 0.153013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156014] ** ** [ 0.159014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161009] ** ** [ 0.163010] ** This means that this kernel is built to expose internal ** [ 0.165029] ** IOMMU data structures, which may compromise security on ** [ 0.167014] ** your system. ** [ 0.169011] ** ** [ 0.172020] ** If you see this message and you are not debugging the ** [ 0.174010] ** kernel, report this immediately to your vendor! ** [ 0.177026] ** ** [ 0.181015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184011] ************************************************************* [ 0.187155] NET: Registered protocol family 16 [ 0.188423] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198108] cpuidle: using governor menu [ 0.199885] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200540] PCI: Using configuration type 1 for base access [ 0.201124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216049] cryptd: max_cpu_qlen set to 1000 [ 0.219371] ACPI: Added _OSI(Module Device) [ 0.221013] ACPI: Added _OSI(Processor Device) [ 0.222014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.229615] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.236501] ACPI: Interpreter enabled [ 0.238038] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239014] ACPI: Using IOAPIC for interrupt routing [ 0.241130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244644] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253681] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.257019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.260103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266611] acpiphp: Slot [2] registered [ 0.269159] acpiphp: Slot [5] registered [ 0.271128] acpiphp: Slot [6] registered [ 0.272170] acpiphp: Slot [7] registered [ 0.274115] acpiphp: Slot [8] registered [ 0.275142] acpiphp: Slot [9] registered [ 0.276106] acpiphp: Slot [10] registered [ 0.277085] acpiphp: Slot [3] registered [ 0.279105] acpiphp: Slot [4] registered [ 0.280074] acpiphp: Slot [11] registered [ 0.281108] acpiphp: Slot [12] registered [ 0.282064] acpiphp: Slot [13] registered [ 0.284084] acpiphp: Slot [14] registered [ 0.286108] acpiphp: Slot [15] registered [ 0.287090] acpiphp: Slot [16] registered [ 0.288123] acpiphp: Slot [17] registered [ 0.290119] acpiphp: Slot [18] registered [ 0.291119] acpiphp: Slot [19] registered [ 0.293106] acpiphp: Slot [20] registered [ 0.295107] acpiphp: Slot [21] registered [ 0.296065] acpiphp: Slot [22] registered [ 0.297096] acpiphp: Slot [23] registered [ 0.298068] acpiphp: Slot [24] registered [ 0.299088] acpiphp: Slot [25] registered [ 0.301104] acpiphp: Slot [26] registered [ 0.302093] acpiphp: Slot [27] registered [ 0.303074] acpiphp: Slot [28] registered [ 0.304100] acpiphp: Slot [29] registered [ 0.306110] acpiphp: Slot [30] registered [ 0.307069] acpiphp: Slot [31] registered [ 0.308059] PCI host bridge to bus 0000:00 [ 0.310019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.318023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.320021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.323027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.325182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.328059] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.331496] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.347059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.349019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357680] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359860] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365685] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.370014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.381015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.386015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.391729] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.407020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.425018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.434542] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.443016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.456018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.486018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.496258] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.507018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.516015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.538021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.550551] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.559017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.567018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.587018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.600589] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.606015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.612014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.634031] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.645951] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.654023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.665017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.689027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.701876] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.703237] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.705346] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.706223] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.707158] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.710146] iommu: Default domain type: Passthrough [ 0.714263] SCSI subsystem initialized [ 0.715195] ACPI: bus type USB registered [ 0.717103] usbcore: registered new interface driver usbfs [ 0.719087] usbcore: registered new interface driver hub [ 0.721115] usbcore: registered new device driver usb [ 0.723136] pps_core: LinuxPPS API ver. 1 registered [ 0.724009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.727067] PTP clock support registered [ 0.729229] EDAC MC: Ver: 3.0.0 [ 0.731842] PCI: Using ACPI for IRQ routing [ 0.733797] NetLabel: Initializing [ 0.735011] NetLabel: domain hash size = 128 [ 0.736009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.738084] NetLabel: unlabeled traffic allowed by default [ 0.740339] vgaarb: loaded [ 0.741198] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.748343] clocksource: Switched to clocksource kvm-clock [ 0.847795] VFS: Disk quotas dquot_6.6.0 [ 0.849379] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.850997] *** VALIDATE ramfs *** [ 0.851728] *** VALIDATE hugetlbfs *** [ 0.853218] pnp: PnP ACPI init [ 0.855454] pnp: PnP ACPI: found 6 devices [ 0.873282] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.876670] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.878831] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.880807] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.883362] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.885748] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.888635] NET: Registered protocol family 2 [ 0.890873] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.895530] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.899404] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.904853] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.908413] TCP: Hash tables configured (established 65536 bind 65536) [ 0.911236] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.914449] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.917357] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920469] NET: Registered protocol family 1 [ 0.923271] RPC: Registered named UNIX socket transport module. [ 0.926037] RPC: Registered udp transport module. [ 0.927937] RPC: Registered tcp transport module. [ 0.930057] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.933074] NET: Registered protocol family 44 [ 0.934851] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.936862] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.939130] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.941146] PCI: CLS 0 bytes, default 64 [ 0.943642] Unpacking initramfs... [ 2.331313] debug: unmapping init [mem 0xffff8e51fcc54000-0xffff8e51fffbffff] [ 2.335245] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.337525] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.340088] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.802970] Initialise system trusted keyrings [ 2.804652] Key type blacklist registered [ 2.806054] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.813424] zbud: loaded [ 2.815580] *** VALIDATE nfs *** [ 2.816405] *** VALIDATE nfs4 *** [ 2.817446] pstore: using deflate compression [ 2.819878] Platform Keyring initialized [ 2.910087] NET: Registered protocol family 38 [ 2.911483] Key type asymmetric registered [ 2.912598] Asymmetric key parser 'x509' registered [ 2.914102] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.917336] io scheduler mq-deadline registered [ 2.918791] io scheduler kyber registered [ 2.920572] io scheduler bfq registered [ 2.922976] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.925878] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.928503] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.931551] ACPI: Power Button [PWRF] [ 2.935481] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.940698] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.952525] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.959261] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.972034] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.999514] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.027155] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.032319] Non-volatile memory driver v1.3 [ 3.034142] Linux agpgart interface v0.103 [ 3.068089] virtio_blk virtio1: [vda] 146024 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.072132] vda: detected capacity change from 0 to 74764288 [ 3.090479] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.093516] vdb: detected capacity change from 0 to 1073741824 [ 3.111769] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.115054] vdc: detected capacity change from 0 to 2621440000 [ 3.133725] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.136630] vdd: detected capacity change from 0 to 2621440000 [ 3.155573] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.158104] vde: detected capacity change from 0 to 4294967296 [ 3.174706] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.177587] vdf: detected capacity change from 0 to 4294967296 [ 3.191423] libphy: Fixed MDIO Bus: probed [ 3.202828] usbcore: registered new interface driver usbserial_generic [ 3.205436] usbserial: USB Serial support registered for generic [ 3.208151] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.212536] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.214824] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.217322] mousedev: PS/2 mouse device common for all mice [ 3.221161] rtc_cmos 00:05: RTC can wake from S4 [ 3.223502] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.225387] rtc_cmos 00:05: registered as rtc0 [ 3.233782] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.237172] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.237659] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.244063] intel_pstate: CPU model not supported [ 3.247316] hid: raw HID events driver (C) Jiri Kosina [ 3.249563] usbcore: registered new interface driver usbhid [ 3.251680] usbhid: USB HID core driver [ 3.253389] drop_monitor: Initializing network drop monitor service [ 3.256190] Initializing XFRM netlink socket [ 3.258388] NET: Registered protocol family 10 [ 3.261962] Segment Routing with IPv6 [ 3.263320] NET: Registered protocol family 17 [ 3.265702] mpls_gso: MPLS GSO support [ 3.270633] RAS: Correctable Errors collector initialized. [ 3.272987] AVX version of gcm_enc/dec engaged. [ 3.274676] AES CTR mode by8 optimization enabled [ 3.351368] sched_clock: Marking stable (3351331516, 0)->(4228939749, -877608233) [ 3.355381] registered taskstats version 1 [ 3.357802] Loading compiled-in X.509 certificates [ 3.360376] zswap: loaded using pool lzo/zbud [ 3.387661] Key type big_key registered [ 3.400691] Key type encrypted registered [ 3.402420] ima: No TPM chip found, activating TPM-bypass! [ 3.404637] ima: Allocated hash algorithm: sha1 [ 3.406433] ima: No architecture policies found [ 3.408369] evm: Initialising EVM extended attributes: [ 3.410263] evm: security.selinux [ 3.411228] evm: security.ima [ 3.412162] evm: security.capability [ 3.413590] evm: HMAC attrs: 0x1 [ 3.416694] rtc_cmos 00:05: setting system clock to 2026-09-02 00:30:35 UTC (1788309035) [ 3.423323] debug: unmapping init [mem 0xffffffff92203000-0xffffffff923fffff] [ 3.426698] debug: unmapping init [mem 0xffffffff90f82000-0xffffffff91258fff] [ 3.434684] Write protecting the kernel read-only data: 28672k [ 3.437962] debug: unmapping init [mem 0xffffffff8f603000-0xffffffff8f7fffff] [ 3.440526] debug: unmapping init [mem 0xffffffff8ff14000-0xffffffff8fffffff] [ 3.474336] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.482508] systemd[1]: Detected virtualization kvm. [ 3.484603] systemd[1]: Detected architecture x86-64. [ 3.486636] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.508283] systemd[1]: No hostname configured. [ 3.510087] systemd[1]: Set hostname to . [ 3.512451] random: systemd: uninitialized urandom read (16 bytes read) [ 3.515182] systemd[1]: Initializing machine ID from random generator. [ 3.657676] random: systemd: uninitialized urandom read (16 bytes read) [ 3.660475] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.664279] random: systemd: uninitialized urandom read (16 bytes read) [ 3.666937] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.672070] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.289970] device-mapper: uevent: version 1.0.3 [ 4.291936] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.050606] random: fast init done [ 5.080675] virtio_net virtio0 ens2: renamed from eth0 [ 5.131340] scsi host0: ata_piix [ 5.200356] scsi host1: ata_piix [ 5.201798] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.203954] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.018810] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.919776] random: crng init done [ 9.921268] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.410801] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.604112] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.912624] SELinux: Disabled at runtime. [ 11.971968] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.979242] systemd[1]: Detected virtualization kvm. [ 11.980635] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.525300] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.529208] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.534189] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.537245] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.539687] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.550477] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.559340] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. [ 12.649036] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.971914] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.236863] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.281719] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.405580] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.423524] EDAC sbridge: Ver: 1.1.2 [ 15.081307] Key type dns_resolver registered [ 15.382820] NFS: Registering the id_resolver key type [ 15.384852] Key type id_resolver registered [ 15.386204] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg102-server login: [ 38.380425] hrtimer: interrupt took 3373990 ns [ 52.854943] libcfs: loading out-of-tree module taints kernel. [ 52.890430] Key type ._llcrypt registered [ 52.892455] Key type .llcrypt registered [ 52.985975] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 72.136270] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 74.309265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 74.327665] alg: No test for adler32 (adler32-zlib) [ 75.842975] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 76.779308] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 78.487240] Key type lgssc registered [ 80.266943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.956681] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 130.765484] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 142.488693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.534536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.771303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.795842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.875827] Lustre: lustre-MDT0000: new disk, initializing [ 143.971875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.989726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.312221] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.998044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.071559] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 159.103461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 159.106962] Lustre: Skipped 1 previous similar message [ 159.176558] Lustre: lustre-MDT0001: new disk, initializing [ 159.238042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.268634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.279814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 163.813554] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.205184] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 176.885924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 177.077810] Lustre: lustre-OST0000: new disk, initializing [ 177.080685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 177.083961] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 177.128960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.278790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.287749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.349160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 182.359773] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.006763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 195.107871] Lustre: lustre-OST0001: new disk, initializing [ 195.111786] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 195.115648] Lustre: 9515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 195.153925] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.047424] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.792257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 202.808692] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 202.854311] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 212.897894] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 219.005172] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 224.612717] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing check_logdir /tmp/testlogs/ [ 229.861476] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing yml_node [ 233.173912] Lustre: DEBUG MARKER: Client: 2.17.57.45 [ 235.329625] Lustre: DEBUG MARKER: MDS: 2.17.57.45 [ 237.446232] Lustre: DEBUG MARKER: OSS: 2.17.57.45 [ 239.072786] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 1 20:34:30 EDT 2026 [ 253.486443] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 254.666800] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 269.284437] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 269.286546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 269.300727] Lustre: Skipped 3 previous similar messages [ 269.308717] Lustre: Skipped 2 previous similar messages [ 274.400110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 274.402697] Lustre: Skipped 1 previous similar message [ 274.958890] Lustre: server umount lustre-MDT0000 complete [ 278.668820] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309310 with bad export cookie 15638660603966113415 [ 278.669450] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 278.678501] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 279.523783] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.532635] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 279.540525] Lustre: Skipped 1 previous similar message [ 283.553141] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.560629] Lustre: Skipped 2 previous similar messages [ 285.059991] Lustre: server umount lustre-MDT0001 complete [ 288.801342] Lustre: server umount lustre-OST0000 complete [ 292.521723] Lustre: server umount lustre-OST0001 complete [ 300.193376] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 309.286961] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 341.019290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 341.188576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 341.209220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 341.251896] Lustre: lustre-MDT0000: new disk, initializing [ 341.315473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 341.348322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 345.269619] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 354.685075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 354.806390] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 354.865495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 354.870683] Lustre: Skipped 1 previous similar message [ 354.976948] Lustre: lustre-MDT0001: new disk, initializing [ 355.105080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 355.141287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 355.149719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 359.026541] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 367.867371] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 372.053747] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 376.864756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 377.111261] Lustre: lustre-OST0000: new disk, initializing [ 377.113944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 377.119026] Lustre: 17035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 377.190278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 378.221693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 378.226224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 378.296060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 381.477512] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 390.549762] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 394.569253] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 394.755211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 399.050868] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 399.220472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 403.938279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.944473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 403.967858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 414.186698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 414.205112] Lustre: Skipped 4 previous similar messages [ 415.199251] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 415.433825] Lustre: server umount lustre-OST0000 complete [ 419.305224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.336258] Lustre: Skipped 2 previous similar messages [ 425.265972] Lustre: server umount lustre-MDT0000 complete [ 429.442717] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309461 with bad export cookie 15638660603966114948 [ 429.478736] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 429.795452] Lustre: server umount lustre-MDT0001 complete [ 433.738668] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 20:37:44 (1788309464) [ 440.429193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.796907] LustreError: 19300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.805534] LustreError: 19300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 440.872632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.714027] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 445.928509] LustreError: 19301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.028657] LustreError: 19300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 453.561228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.381946] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 464.634843] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 470.218770] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 476.209534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.332218] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 487.874624] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 496.097036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.100465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.105863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.115680] Lustre: Skipped 1 previous similar message [ 496.129276] Lustre: Skipped 5 previous similar messages [ 500.913732] Lustre: server umount lustre-OST0000 complete [ 505.315047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 505.324396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.459798] LustreError: 19295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 511.486782] LustreError: 19295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 512.736914] Lustre: server umount lustre-MDT0000 complete [ 516.262654] LustreError: 19281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309548 with bad export cookie 15638660603966115823 [ 516.266115] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.269481] LustreError: 19281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 516.553635] Lustre: server umount lustre-MDT0001 complete [ 527.928447] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 530.828796] Key type lgssc unregistered [ 531.055763] LNet: 22835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 531.069986] LNetError: 22835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 531.090543] LNet: Removed LNI 192.168.201.102@tcp [ 531.817176] Key type .llcrypt unregistered [ 531.820414] Key type ._llcrypt unregistered [ 557.349333] Key type ._llcrypt registered [ 557.351608] Key type .llcrypt registered [ 557.503514] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 558.911490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 559.186543] alg: No test for adler32 (adler32-zlib) [ 560.182518] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 560.428094] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 562.144227] Key type lgssc registered [ 563.103905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 570.292695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 570.305179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.592165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.968463] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 581.110161] LustreError: 24246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 581.132350] LustreError: 24246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 583.612396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.041480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 588.529344] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 595.490184] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 602.015684] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 608.490480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 608.806465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 615.113638] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 616.057290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 621.970865] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 636.898241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 636.901815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 636.907990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 636.917109] Lustre: Skipped 1 previous similar message [ 636.929644] Lustre: Skipped 1 previous similar message [ 638.982564] Lustre: server umount lustre-OST0000 complete [ 645.600282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 645.608858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 645.625571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.142863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.144765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 651.482737] Lustre: server umount lustre-MDT0000 complete [ 652.259698] LustreError: 24247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.275051] LustreError: 24247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 655.026416] LustreError: 24227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309687 with bad export cookie 1430538922870189725 [ 655.040079] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 655.417823] Lustre: server umount lustre-MDT0001 complete [ 666.135741] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 668.588929] Key type lgssc unregistered [ 668.943472] LNet: 28226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.957466] LNetError: 28226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.985687] LNet: Removed LNI 192.168.201.102@tcp [ 669.906215] Key type .llcrypt unregistered [ 669.909475] Key type ._llcrypt unregistered [ 695.212672] Key type ._llcrypt registered [ 695.214480] Key type .llcrypt registered [ 695.330625] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 697.477241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 697.527670] alg: No test for adler32 (adler32-zlib) [ 698.593415] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 698.853743] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 700.535430] Key type lgssc registered [ 701.621574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 711.419968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 711.456878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.879648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.954247] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 722.405053] LustreError: 29644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 722.439084] LustreError: 29644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 725.877839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.314560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 730.469424] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 737.907179] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 743.364757] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 757.312375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 757.811085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 764.011580] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 772.931678] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 782.383847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 791.521907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 791.548680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.580248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 792.557028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 792.574102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.575763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 793.578405] Lustre: server umount lustre-OST0000 complete [ 802.793620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.833253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 802.835979] Lustre: Skipped 1 previous similar message [ 805.971391] Lustre: server umount lustre-MDT0000 complete [ 807.907341] LustreError: 29644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 807.938379] LustreError: 29644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 810.717061] LustreError: 29624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309842 with bad export cookie 8527440357095391872 [ 810.719945] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 810.729166] LustreError: 29624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 811.263544] Lustre: server umount lustre-MDT0001 complete [ 823.994202] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 826.543500] Key type lgssc unregistered [ 826.816990] LNet: 33547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.822214] LNetError: 33547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.831812] LNet: Removed LNI 192.168.201.102@tcp [ 827.596991] Key type .llcrypt unregistered [ 827.599535] Key type ._llcrypt unregistered [ 847.076107] Key type ._llcrypt registered [ 847.077946] Key type .llcrypt registered [ 847.189329] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 848.276896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 848.320452] alg: No test for adler32 (adler32-zlib) [ 849.378824] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 849.630896] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 851.415199] Key type lgssc registered [ 852.292731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 859.395605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 859.409382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.889230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 865.137356] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 870.372767] LustreError: 34939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.402423] LustreError: 34939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 873.385295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 873.743758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 877.863648] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 885.306645] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 890.581829] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 896.835280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.243919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 903.231723] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 905.465790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 911.045573] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 919.014337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 919.022878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.040788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 921.074298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 921.083563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.114176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 926.182224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 926.191675] Lustre: Skipped 1 previous similar message [ 931.306813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 931.312129] Lustre: Skipped 2 previous similar messages [ 932.319315] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 932.435564] Lustre: server umount lustre-OST0000 complete [ 936.418257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 936.425728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 936.455586] Lustre: Skipped 1 previous similar message [ 946.659292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.676184] Lustre: Skipped 4 previous similar messages [ 950.239189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 950.334641] Lustre: server umount lustre-MDT0000 complete [ 951.785126] LustreError: 34938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.812191] LustreError: 34938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 953.317841] LustreError: 34921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788309985 with bad export cookie 9766207993437393209 [ 953.318917] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.328893] LustreError: 34921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 953.678484] Lustre: server umount lustre-MDT0001 complete [ 964.917307] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 968.565519] Key type lgssc unregistered [ 969.109591] LNet: 38478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.120768] LNetError: 38478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.149877] LNet: Removed LNI 192.168.201.102@tcp [ 970.334575] Key type .llcrypt unregistered [ 970.336761] Key type ._llcrypt unregistered [ 996.991360] Key type ._llcrypt registered [ 996.995120] Key type .llcrypt registered [ 997.135641] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 998.469973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.606212] alg: No test for adler32 (adler32-zlib) [ 999.770955] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1000.071228] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 1001.791186] Key type lgssc registered [ 1002.869784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1010.371035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1010.397650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.828083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.733341] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1021.924990] LustreError: 39894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1021.953326] LustreError: 39894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1025.574711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.996670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1030.790601] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1038.776702] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1045.200419] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1051.318243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1051.568326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.537827] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1057.713057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1065.235794] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1075.174491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.182123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1075.187976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1075.194547] Lustre: Skipped 1 previous similar message [ 1076.498480] Lustre: server umount lustre-OST0000 complete [ 1097.696824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1097.706575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1097.719786] Lustre: Skipped 1 previous similar message [ 1097.729074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1100.773235] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.781292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1102.197357] Lustre: server umount lustre-MDT0000 complete [ 1105.894597] LustreError: 40895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.265703] LustreError: 39874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310138 with bad export cookie 11055828863042372737 [ 1106.279766] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1106.288309] LustreError: 39874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1112.767394] Lustre: server umount lustre-MDT0001 complete [ 1124.664559] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 1128.641789] Key type lgssc unregistered [ 1129.181679] LNet: 43606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.185838] LNetError: 43606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.213166] LNet: Removed LNI 192.168.201.102@tcp [ 1130.398473] Key type .llcrypt unregistered [ 1130.400116] Key type ._llcrypt unregistered [ 1158.746673] Key type ._llcrypt registered [ 1158.749884] Key type .llcrypt registered [ 1158.832108] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 1160.945823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1161.025591] alg: No test for adler32 (adler32-zlib) [ 1162.083739] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1162.348949] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 1164.095290] Key type lgssc registered [ 1164.740883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1172.042475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1172.087720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1173.419467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1177.264282] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1183.716408] LustreError: 45025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.728074] LustreError: 45025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1185.430867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.860899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1190.551734] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1197.948069] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1203.331279] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1209.832621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1210.193990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.501947] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1218.432757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1223.625120] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1231.233634] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.2@tcp (stopping) [ 1231.844151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1231.850277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.872440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.908497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.916327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.928472] Lustre: Skipped 1 previous similar message [ 1233.949154] Lustre: Skipped 2 previous similar messages [ 1236.333501] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.2@tcp (stopping) [ 1236.830945] Lustre: server umount lustre-MDT0000 complete [ 1239.015677] LustreError: 46235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.044536] LustreError: 46235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1240.649964] LustreError: 46869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310272 with bad export cookie 11256194877427617124 [ 1240.657588] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.675713] LustreError: 46869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1241.447859] LustreError: 45020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.498164] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.2@tcp (stopping) [ 1244.131581] LustreError: 45021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.133849] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.084760] Lustre: server umount lustre-MDT0001 complete [ 1267.954550] Lustre: server umount lustre-OST0000 complete [ 1283.298058] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 1285.549603] Key type lgssc unregistered [ 1285.788864] LNet: 48887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.802366] LNetError: 48887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.824331] LNet: Removed LNI 192.168.201.102@tcp [ 1286.511251] Key type .llcrypt unregistered [ 1286.513328] Key type ._llcrypt unregistered [ 1306.725471] Key type ._llcrypt registered [ 1306.727830] Key type .llcrypt registered [ 1306.823669] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 1307.921618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1307.946654] alg: No test for adler32 (adler32-zlib) [ 1309.032333] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1309.277680] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 1310.935189] Key type lgssc registered [ 1311.820669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1318.901718] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1318.912631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1334.432395] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1339.551360] LustreError: 50253:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.102@tcp: failed processing log, type 4: rc = -110 [ 1366.239248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.334664] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1381.884880] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1550.189657] Lustre: server umount lustre-OST0000 complete [ 1567.701678] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 1571.439979] Key type lgssc unregistered [ 1571.825657] LNet: 51933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.833379] LNetError: 51933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.909660] LNet: Removed LNI 192.168.201.102@tcp [ 1573.690103] Key type .llcrypt unregistered [ 1573.691662] Key type ._llcrypt unregistered [ 1599.980444] Key type ._llcrypt registered [ 1599.998648] Key type .llcrypt registered [ 1600.144790] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 1601.522992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1601.558677] alg: No test for adler32 (adler32-zlib) [ 1602.786677] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1603.146588] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 1604.871194] Key type lgssc registered [ 1606.173957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1614.843099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1614.859857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.325491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.749548] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1626.610506] LustreError: 53349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.628030] LustreError: 53349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1631.342626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.827198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1636.831962] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1644.698409] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1650.573422] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1657.826975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1658.132576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1661.264463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1664.348276] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1673.079576] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1676.768647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1676.803358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.894559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1681.904193] Lustre: Skipped 1 previous similar message [ 1687.010570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.016377] Lustre: Skipped 1 previous similar message [ 1690.083806] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1690.348916] Lustre: server umount lustre-OST0000 complete [ 1697.251776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.270266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1697.283572] Lustre: Skipped 2 previous similar messages [ 1701.289489] Lustre: server umount lustre-MDT0000 complete [ 1702.369827] LustreError: 53344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.392205] LustreError: 53344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1705.217572] LustreError: 53329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310737 with bad export cookie 3371313456616334164 [ 1705.219616] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1705.230173] LustreError: 53329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1705.795496] Lustre: server umount lustre-MDT0001 complete [ 1716.399076] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 1719.395572] Key type lgssc unregistered [ 1719.764090] LNet: 56883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.769990] LNetError: 56883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.793634] LNet: Removed LNI 192.168.201.102@tcp [ 1720.794405] Key type .llcrypt unregistered [ 1720.796337] Key type ._llcrypt unregistered [ 1741.193554] Key type ._llcrypt registered [ 1741.196537] Key type .llcrypt registered [ 1741.309695] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 1742.191392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1742.258135] alg: No test for adler32 (adler32-zlib) [ 1743.350418] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1743.598883] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 1745.256051] Key type lgssc registered [ 1746.674125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1753.942063] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1753.960206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1769.439884] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1774.562690] LustreError: 58248:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.102@tcp: failed processing log, type 4: rc = -110 [ 1801.183239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1810.388435] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1820.110055] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1827.954973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.872670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.045754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1833.002421] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1833.953846] LustreError: 58273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1833.969360] LustreError: 58273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1836.009738] LustreError: 59286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1836.020405] LustreError: 59286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1841.124613] LustreError: 59284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1841.150231] LustreError: 59284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1841.587108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.789679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1846.218575] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1855.652913] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1861.581226] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1864.677813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1864.684333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.700616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.235498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.238387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.273932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1870.207393] Lustre: server umount lustre-OST0000 complete [ 1897.457963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.487770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.490264] Lustre: Skipped 1 previous similar message [ 1897.510134] Lustre: Skipped 2 previous similar messages [ 1900.152332] Lustre: server umount lustre-MDT0000 complete [ 1902.562190] LustreError: 59290:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1902.578653] LustreError: 59290:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1904.762311] LustreError: 58256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788310936 with bad export cookie 7216415452659727566 [ 1904.773123] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.248750] Lustre: server umount lustre-MDT0001 complete [ 1924.174700] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 1927.776486] Key type lgssc unregistered [ 1928.257818] LNet: 62012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.277379] LNetError: 62012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.314749] LNet: Removed LNI 192.168.201.102@tcp [ 1929.414828] Key type .llcrypt unregistered [ 1929.416599] Key type ._llcrypt unregistered [ 1953.497713] Key type ._llcrypt registered [ 1953.500313] Key type .llcrypt registered [ 1953.661056] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 1955.276854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.377803] alg: No test for adler32 (adler32-zlib) [ 1956.392782] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 1956.660349] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 1958.335239] Key type lgssc registered [ 1959.358921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1967.313586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1967.333949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.871550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1973.974893] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1978.358426] LustreError: 63427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.385604] LustreError: 63427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1983.033543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.337181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1987.801255] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1995.484443] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2000.588284] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2007.187274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2007.758597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.007657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2014.884585] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2022.384326] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2043.879387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2043.887302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.900667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2043.915837] Lustre: Skipped 1 previous similar message [ 2046.718903] Lustre: server umount lustre-OST0000 complete [ 2054.112725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2054.115781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2054.133167] Lustre: Skipped 1 previous similar message [ 2054.142000] Lustre: Skipped 2 previous similar messages [ 2057.492791] Lustre: server umount lustre-MDT0000 complete [ 2059.232091] LustreError: 64155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.254814] LustreError: 64155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2060.999535] LustreError: 63406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311093 with bad export cookie 12515035524878148768 [ 2061.007855] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2061.012561] LustreError: 63406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2061.296894] Lustre: server umount lustre-MDT0001 complete [ 2071.449514] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 2073.684872] Key type lgssc unregistered [ 2073.878978] LNet: 66983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.885276] LNetError: 66983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.898899] LNet: Removed LNI 192.168.201.102@tcp [ 2074.620637] Key type .llcrypt unregistered [ 2074.622530] Key type ._llcrypt unregistered [ 2098.035368] Key type ._llcrypt registered [ 2098.039409] Key type .llcrypt registered [ 2098.145112] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2099.262068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.339093] alg: No test for adler32 (adler32-zlib) [ 2100.414780] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2100.721759] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 2102.376946] Key type lgssc registered [ 2103.492917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2113.556031] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 21:05:44 (1788311144) [ 2120.877727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2120.903615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.474417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.247494] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2131.959907] LustreError: 68680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.986868] LustreError: 68680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2136.205780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.478843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2141.106727] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2147.558685] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2153.153909] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2160.435410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.712426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2166.982552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2167.080382] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2175.201812] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2181.131182] Lustre: Failing over lustre-MDT0000 [ 2181.361960] Lustre: server umount lustre-MDT0000 complete [ 2182.627777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2182.639581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.653037] LustreError: 68681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.707980] Lustre: Skipped 1 previous similar message [ 2187.747752] LustreError: 68676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.757291] LustreError: 68676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2191.740300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.992493] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2192.273647] Lustre: *** cfs_fail_loc=135, val=0*** [ 2192.499273] LustreError: 71768:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2192.512826] LustreError: 71768:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.102@tcp: cfg command failed: rc = -2 [ 2192.530859] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2192.530859] [ 2192.553705] LustreError: MGC192.168.201.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2192.596766] LustreError: 71755:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2192.622860] LustreError: 71755:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2192.652781] LustreError: 71755:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2192.667117] LustreError: 71755:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2192.755507] Lustre: server umount lustre-MDT0000 complete [ 2192.761331] LustreError: 71755:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2192.877350] LustreError: 68680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.921286] LustreError: 68680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2197.987710] LustreError: 69408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.012703] LustreError: 69408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2199.955765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.103521] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2200.367982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.421225] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2203.522469] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2205.698850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2205.761954] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2205.826167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2207.152183] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2215.906463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.908525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2215.910747] Lustre: Skipped 2 previous similar messages [ 2221.029160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2221.039110] Lustre: Skipped 2 previous similar messages [ 2221.576609] Lustre: server umount lustre-OST0000 complete [ 2226.151956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2226.159774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.163682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2226.207182] Lustre: Skipped 1 previous similar message [ 2231.266412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.275584] Lustre: Skipped 2 previous similar messages [ 2232.252916] Lustre: server umount lustre-MDT0000 complete [ 2235.870929] LustreError: 68661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311267 with bad export cookie 13123962146663413147 [ 2235.872158] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2235.880787] LustreError: 68661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2236.373561] Lustre: server umount lustre-MDT0001 complete [ 2246.903590] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 2249.535079] Key type lgssc unregistered [ 2249.771546] LNet: 73552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.777876] LNetError: 73552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.791963] LNet: Removed LNI 192.168.201.102@tcp [ 2250.600357] Key type .llcrypt unregistered [ 2250.603259] Key type ._llcrypt unregistered [ 2276.769682] Key type ._llcrypt registered [ 2276.771200] Key type .llcrypt registered [ 2276.927206] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2277.813087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2277.847600] alg: No test for adler32 (adler32-zlib) [ 2278.918915] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2279.372508] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 2281.087211] Key type lgssc registered [ 2282.545518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2290.422547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2290.452550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.835775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2296.685410] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2301.426045] LustreError: 74966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.452059] LustreError: 74966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2304.877964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.298296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2309.837816] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2316.240776] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2320.432171] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2326.037102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.329301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.203343] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2336.648226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2339.432278] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2344.616043] Lustre: Failing over lustre-MDT0000 [ 2344.791942] Lustre: server umount lustre-MDT0000 complete [ 2346.376359] LustreError: 75988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.465484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2346.468618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.978841] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2346.994466] Lustre: Skipped 1 previous similar message [ 2351.465797] LustreError: 74961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2351.481369] LustreError: 74961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2352.952543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.063932] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2353.205469] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2353.403806] LustreError: 78052:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2353.420878] LustreError: 78052:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.102@tcp: cfg command failed: rc = -9 [ 2353.435644] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2353.435644] [ 2353.463106] LustreError: MGC192.168.201.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2353.490354] LustreError: 78040:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2353.505929] LustreError: 78040:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2353.536597] LustreError: 78040:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2353.553953] LustreError: 78040:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2353.598683] Lustre: server umount lustre-MDT0000 complete [ 2353.600593] LustreError: 78040:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2356.595867] LustreError: 75988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.622321] LustreError: 75988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2361.709093] LustreError: 74962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.747324] LustreError: 74962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2361.863907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.990546] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.157882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.219581] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2366.829184] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2367.037836] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2367.463930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2367.519544] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2367.606095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2376.672101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2376.675978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.684762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.697832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.712271] Lustre: Skipped 1 previous similar message [ 2380.807352] Lustre: server umount lustre-OST0000 complete [ 2387.941791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2387.942220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.956040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2387.988368] Lustre: Skipped 2 previous similar messages [ 2393.058347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.061216] Lustre: Skipped 1 previous similar message [ 2398.179072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2398.188272] Lustre: Skipped 1 previous similar message [ 2399.199554] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2399.408512] Lustre: server umount lustre-MDT0000 complete [ 2402.280775] LustreError: 74965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.314593] LustreError: 74965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2403.610387] LustreError: 74947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311435 with bad export cookie 3172147173087148366 [ 2403.622851] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2404.044918] Lustre: server umount lustre-MDT0001 complete [ 2416.540246] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 2419.892441] Key type lgssc unregistered [ 2420.228723] LNet: 79838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.236838] LNetError: 79838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.260743] LNet: Removed LNI 192.168.201.102@tcp [ 2421.175808] Key type .llcrypt unregistered [ 2421.177299] Key type ._llcrypt unregistered [ 2434.652870] Key type ._llcrypt registered [ 2434.655181] Key type .llcrypt registered [ 2434.756726] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 2435.058427] Key type .llcrypt unregistered [ 2435.060058] Key type ._llcrypt unregistered [ 2459.201802] Key type ._llcrypt registered [ 2459.203798] Key type .llcrypt registered [ 2459.315453] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2461.018354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.233635] alg: No test for adler32 (adler32-zlib) [ 2462.342216] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2462.639652] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 2464.400743] Key type lgssc registered [ 2465.827196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2473.901971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2473.923260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.334982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.223259] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2485.735253] LustreError: 81747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.748933] LustreError: 81747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2487.798184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.097781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2492.336608] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2499.643868] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2504.665761] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2511.446464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2511.701964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.853976] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2520.956772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2525.797096] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2534.369168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2534.378350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2534.396440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2536.420396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2536.422242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2536.428733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.445442] Lustre: Skipped 1 previous similar message [ 2540.187993] Lustre: server umount lustre-OST0000 complete [ 2544.608539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2544.614982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.635829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2546.658058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2546.660373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.244878] Lustre: server umount lustre-MDT0000 complete [ 2551.776075] LustreError: 82475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.445083] LustreError: 81728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311585 with bad export cookie 2853265105402768420 [ 2553.449312] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.462298] LustreError: 81728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2553.738226] Lustre: server umount lustre-MDT0001 complete [ 2565.183515] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 2568.285792] Key type lgssc unregistered [ 2568.641082] LNet: 85283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.646571] LNetError: 85283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2568.662612] LNet: Removed LNI 192.168.201.102@tcp [ 2569.679104] Key type .llcrypt unregistered [ 2569.688142] Key type ._llcrypt unregistered [ 2593.909511] Key type ._llcrypt registered [ 2593.911600] Key type .llcrypt registered [ 2594.017693] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2595.401034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2595.439327] alg: No test for adler32 (adler32-zlib) [ 2596.558392] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2596.859435] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 2598.575216] Key type lgssc registered [ 2600.059858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2607.676205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2607.690649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.396163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.466570] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2619.880693] LustreError: 86697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.901357] LustreError: 86697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2623.640275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.022845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2627.829485] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2635.339393] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2643.066744] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2652.416110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2652.857756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2654.918892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2660.552529] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2669.267756] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2679.778648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.795394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2682.777847] Lustre: server umount lustre-OST0000 complete [ 2690.021330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.024918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2690.029850] Lustre: Skipped 2 previous similar messages [ 2690.038443] Lustre: Skipped 2 previous similar messages [ 2695.137677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.152764] Lustre: Skipped 1 previous similar message [ 2699.234175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.239243] Lustre: Skipped 1 previous similar message [ 2701.279169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2701.420949] Lustre: server umount lustre-MDT0000 complete [ 2704.358872] LustreError: 87756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.369855] LustreError: 87756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2704.696276] LustreError: 86677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311736 with bad export cookie 2998371331036298402 [ 2704.711531] LustreError: 86677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2704.712575] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.002830] Lustre: server umount lustre-MDT0001 complete [ 2716.203353] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 2718.989143] Key type lgssc unregistered [ 2719.250799] LNet: 90235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2719.285554] LNetError: 90235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2719.308323] LNet: Removed LNI 192.168.201.102@tcp [ 2720.186626] Key type .llcrypt unregistered [ 2720.188825] Key type ._llcrypt unregistered [ 2740.857396] Key type ._llcrypt registered [ 2740.860914] Key type .llcrypt registered [ 2740.945227] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2741.476323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2741.568797] alg: No test for adler32 (adler32-zlib) [ 2742.657854] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2742.933366] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 2744.719191] Key type lgssc registered [ 2745.421858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2753.215827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2753.241056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.642924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2758.943334] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2764.778264] LustreError: 91646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.804879] LustreError: 91646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2768.135413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.437136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2773.198693] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2780.919289] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2787.064920] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2794.582173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2795.051372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2799.155425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2801.995271] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2810.357075] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2823.142479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2823.150944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.157591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2823.656843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2823.663574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2823.698601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2824.142410] Lustre: server umount lustre-OST0000 complete [ 2828.772094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.792569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2828.797522] Lustre: Skipped 1 previous similar message [ 2833.689939] Lustre: server umount lustre-MDT0000 complete [ 2833.888194] LustreError: 91645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.895733] LustreError: 91645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2836.826781] LustreError: 91627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788311868 with bad export cookie 9280335484197959024 [ 2836.832267] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2836.834387] LustreError: 91627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2837.256866] Lustre: server umount lustre-MDT0001 complete [ 2848.153273] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 2851.211905] Key type lgssc unregistered [ 2851.618943] LNet: 95181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2851.635410] LNetError: 95181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2851.654382] LNet: Removed LNI 192.168.201.102@tcp [ 2852.805719] Key type .llcrypt unregistered [ 2852.809075] Key type ._llcrypt unregistered [ 2873.356630] Key type ._llcrypt registered [ 2873.359127] Key type .llcrypt registered [ 2873.469174] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2874.770955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2874.787063] alg: No test for adler32 (adler32-zlib) [ 2875.906758] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 2876.149072] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 2877.791176] Key type lgssc registered [ 2878.738174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2885.824637] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2885.859617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2901.407336] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2906.527563] LustreError: 96547:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.102@tcp: failed processing log, type 4: rc = -110 [ 2933.279956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.229676] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2946.403712] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2951.002940] Lustre: server umount lustre-OST0000 complete [ 2958.104643] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 21:19:49 (1788311989) [ 2963.581556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.043278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.532196] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2969.062215] LustreError: 98184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.085338] LustreError: 98184:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2974.184290] LustreError: 98185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2976.491078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2976.937391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2981.782337] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2989.169898] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2994.114482] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3000.750570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3001.029625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.580943] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3009.198051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3013.932386] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3027.939504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3027.942564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.963387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3029.985355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3030.020720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3030.034945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.375965] Lustre: server umount lustre-OST0000 complete [ 3038.182482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3038.191531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.215271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3038.225420] Lustre: Skipped 1 previous similar message [ 3040.225085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3040.226152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3040.263773] Lustre: Skipped 1 previous similar message [ 3042.600485] Lustre: server umount lustre-MDT0000 complete [ 3045.346437] LustreError: 98179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3045.362034] LustreError: 98179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3045.598748] LustreError: 100837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312077 with bad export cookie 4553260115769520846 [ 3045.604583] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.610313] LustreError: 100837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3046.366267] Lustre: server umount lustre-MDT0001 complete [ 3055.624677] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 3058.011962] Key type lgssc unregistered [ 3058.232066] LNet: 101878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3058.237879] LNetError: 101878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3058.261412] LNet: Removed LNI 192.168.201.102@tcp [ 3059.083482] Key type .llcrypt unregistered [ 3059.085607] Key type ._llcrypt unregistered [ 3092.373193] Key type ._llcrypt registered [ 3092.375577] Key type .llcrypt registered [ 3092.447974] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 3093.390067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3093.417721] alg: No test for adler32 (adler32-zlib) [ 3094.435236] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 3094.648616] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 3096.303274] Key type lgssc registered [ 3097.273684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3104.489724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3104.511402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3106.059908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3109.607557] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3116.533858] LustreError: 104045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.545427] LustreError: 104045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3117.080190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.336970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3121.784140] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3128.567614] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3133.810630] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3141.332174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3141.788801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3146.942609] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3153.073552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3154.743614] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3163.617553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.632204] Lustre: Skipped 1 previous similar message [ 3163.650381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3168.740400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3168.746887] Lustre: Skipped 1 previous similar message [ 3169.063729] Lustre: server umount lustre-OST0000 complete [ 3173.860541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.864456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3173.864816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3173.866113] Lustre: Skipped 1 previous similar message [ 3173.875104] Lustre: Skipped 2 previous similar messages [ 3178.976786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3178.990024] Lustre: Skipped 1 previous similar message [ 3179.704147] Lustre: server umount lustre-MDT0000 complete [ 3184.011439] LustreError: 104027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312216 with bad export cookie 15295257777918020654 [ 3184.027809] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3184.489896] Lustre: server umount lustre-MDT0001 complete [ 3200.808422] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 3203.975371] Key type lgssc unregistered [ 3204.229159] LNet: 107580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3204.236356] LNetError: 107580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3205.289218] LNet: Removed LNI 192.168.201.102@tcp [ 3206.264652] Key type .llcrypt unregistered [ 3206.267096] Key type ._llcrypt unregistered [ 3229.360693] Key type ._llcrypt registered [ 3229.363127] Key type .llcrypt registered [ 3229.529171] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 3230.883506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3230.907774] alg: No test for adler32 (adler32-zlib) [ 3232.364618] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 3232.679597] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 3234.511251] Key type lgssc registered [ 3235.961984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3243.506402] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3243.518349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3259.167448] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3264.287332] LustreError: 108829:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.102@tcp: failed processing log, type 4: rc = -110 [ 3290.976585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3296.918522] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3304.833772] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3311.806988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.958622] LustreError: 108164:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b9c7e7f8 x1875181320538368/t0(0) o250->MGC192.168.201.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3312.141128] LustreError: MGC192.168.201.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3312.150322] LustreError: 109853:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3312.153609] LustreError: 109853:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3312.158358] LustreError: 109853:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3312.251578] Lustre: server umount lustre-MDT0000 complete [ 3312.255668] LustreError: 109853:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3329.180942] Lustre: server umount lustre-OST0000 complete [ 3338.975715] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 3347.646089] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 3382.547405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3382.791958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3382.826282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3382.921375] Lustre: lustre-MDT0000: new disk, initializing [ 3382.999086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3383.014745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3386.923504] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3397.134450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.293411] Lustre: 112645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3397.370794] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3397.376126] Lustre: Skipped 1 previous similar message [ 3397.478794] Lustre: lustre-MDT0001: new disk, initializing [ 3397.621321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3397.675086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3397.691862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3401.367837] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3410.976558] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3416.781325] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3424.063396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3424.364233] Lustre: lustre-OST0000: new disk, initializing [ 3424.367282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3424.373506] Lustre: 114612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3424.451023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3431.003497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3431.011715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3431.135156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3431.253373] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3440.198432] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3446.217912] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3446.471703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3452.154755] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3452.497341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3456.994586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3456.996367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3456.998184] LustreError: Skipped 1 previous similar message [ 3456.998933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3457.011252] Lustre: Skipped 1 previous similar message [ 3460.079246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3460.085932] Lustre: Skipped 2 previous similar messages [ 3460.607666] Lustre: server umount lustre-OST0000 complete [ 3465.186857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.207765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3465.215472] Lustre: Skipped 1 previous similar message [ 3470.307743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3470.312057] Lustre: Skipped 1 previous similar message [ 3470.854965] Lustre: server umount lustre-MDT0000 complete [ 3474.729900] LustreError: 112636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312506 with bad export cookie 5256388205577711939 [ 3474.739037] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3474.744972] LustreError: 112636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3475.072780] Lustre: server umount lustre-MDT0001 complete [ 3483.087694] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 21:28:34 (1788312514) [ 3486.092861] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3501.800900] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 3511.291933] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 3544.717415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.949829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3544.989835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3545.062698] Lustre: lustre-MDT0000: new disk, initializing [ 3545.150145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3545.164283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3548.976291] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3558.406482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.579181] Lustre: 119515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3558.673413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3558.689104] Lustre: Skipped 1 previous similar message [ 3558.738818] Lustre: lustre-MDT0001: new disk, initializing [ 3558.815255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3558.850952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3558.861394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3563.834959] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3573.431899] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3578.666519] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3584.779406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3585.063722] Lustre: lustre-OST0000: new disk, initializing [ 3585.067684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3585.073600] Lustre: 121479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3585.174666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.062540] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3595.300947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3595.317545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3595.465882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3599.575671] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3604.231307] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3604.441066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3608.527561] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3608.786155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3611.106808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3611.109722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.120661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3611.151365] Lustre: Skipped 2 previous similar messages [ 3616.639538] Lustre: server umount lustre-OST0000 complete [ 3620.323425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3620.333333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.352568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3620.355183] Lustre: Skipped 3 previous similar messages [ 3626.729315] Lustre: server umount lustre-MDT0000 complete [ 3630.568219] LustreError: 119506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312662 with bad export cookie 5256388205577712814 [ 3630.569576] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3630.576298] LustreError: 119506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3631.069709] Lustre: server umount lustre-MDT0001 complete [ 3640.278792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.825927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.835509] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3645.937982] LustreError: 123697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.967393] LustreError: 123697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3650.023865] LustreError: 123698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.246750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.759550] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3667.412200] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3673.587129] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3680.579243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3686.795133] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3695.554305] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3705.368187] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3711.461121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3711.466228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.481803] Lustre: Skipped 1 previous similar message [ 3711.491282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3711.495699] Lustre: Skipped 4 previous similar messages [ 3716.509718] Lustre: server umount lustre-OST0000 complete [ 3721.196724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3721.203343] LustreError: Skipped 1 previous similar message [ 3721.205198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.224381] Lustre: Skipped 1 previous similar message [ 3726.756771] Lustre: server umount lustre-MDT0000 complete [ 3729.890445] LustreError: 126198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3729.912203] LustreError: 126198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3730.069341] LustreError: 123677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312762 with bad export cookie 5256388205577713696 [ 3730.074532] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3730.076052] LustreError: 123677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3730.409722] Lustre: server umount lustre-MDT0001 complete [ 3744.693098] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 3747.981416] Key type lgssc unregistered [ 3748.361971] LNet: 127339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3748.369249] LNetError: 127339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3748.383239] LNet: Removed LNI 192.168.201.102@tcp [ 3749.200661] Key type .llcrypt unregistered [ 3749.208966] Key type ._llcrypt unregistered [ 3767.780223] Key type ._llcrypt registered [ 3767.784439] Key type .llcrypt registered [ 3767.891665] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 3787.150603] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 3789.191365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3789.239405] alg: No test for adler32 (adler32-zlib) [ 3790.510390] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 3791.080169] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 3792.887959] Key type lgssc registered [ 3794.423818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3831.483860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3831.518296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3832.824268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3832.844787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3832.939273] Lustre: lustre-MDT0000: new disk, initializing [ 3833.044321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3833.076762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3839.380483] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3850.680446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3850.816745] Lustre: 130495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3850.874802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3850.879097] Lustre: Skipped 1 previous similar message [ 3850.950405] Lustre: lustre-MDT0001: new disk, initializing [ 3851.062043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3851.100744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3851.110133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3855.539990] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3864.832189] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3870.600893] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3878.155311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3878.394449] Lustre: lustre-OST0000: new disk, initializing [ 3878.397696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3878.402234] Lustre: 132466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3878.477633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3884.873586] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3886.700590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3886.715235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3886.825776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3895.040758] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3899.592578] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3899.748223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3903.789837] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3904.076578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3907.567668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3907.573715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.576286] LustreError: Skipped 1 previous similar message [ 3907.580770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3907.612733] Lustre: Skipped 1 previous similar message [ 3912.672810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3912.685671] Lustre: Skipped 2 previous similar messages [ 3917.799587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3917.806476] Lustre: Skipped 1 previous similar message [ 3920.351127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3920.710188] Lustre: server umount lustre-OST0000 complete [ 3928.035173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3928.035449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.047596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3928.065124] Lustre: Skipped 1 previous similar message [ 3931.137752] Lustre: server umount lustre-MDT0000 complete [ 3935.135256] LustreError: 130488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788312967 with bad export cookie 9278452807037518932 [ 3935.141985] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3935.150671] LustreError: 130488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3935.542831] Lustre: server umount lustre-MDT0001 complete [ 3945.188978] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 21:36:16 (1788312976) [ 3951.744548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.087576] LustreError: 134880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.105384] LustreError: 134880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3952.198691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3956.761592] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3957.216874] LustreError: 134881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3962.342761] LustreError: 134880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3964.486152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.775211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3969.051877] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3976.777910] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3983.006994] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3984.993345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.012403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3985.022349] Lustre: Skipped 2 previous similar messages [ 3991.342685] Lustre: server umount lustre-MDT0000 complete [ 3994.592297] LustreError: 135184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3995.346570] LustreError: 134862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313027 with bad export cookie 9278452807037519814 [ 3995.366304] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3995.759727] Lustre: server umount lustre-MDT0001 complete [ 4004.513332] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 21:37:15 (1788313035) [ 4011.058913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4026.720213] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4031.903591] LustreError: 137292:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.102@tcp: failed processing log, type 4: rc = -110 [ 4057.631551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.703409] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4071.960485] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4074.964948] Lustre: server umount lustre-OST0000 complete [ 4083.609365] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 21:38:34 (1788313114) [ 4091.788850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4092.716113] LustreError: 138741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4092.752123] LustreError: 138741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4092.886344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.989755] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4102.120313] LustreError: 138742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4102.137440] LustreError: 138742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4109.520950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.294214] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4127.135028] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4135.757618] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4143.830328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4144.221855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4144.232467] Lustre: Skipped 1 previous similar message [ 4151.674443] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4160.529626] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4176.353600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4176.371826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4176.401668] Lustre: Skipped 1 previous similar message [ 4176.418434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4176.427685] Lustre: Skipped 2 previous similar messages [ 4177.438816] Lustre: server umount lustre-MDT0000 complete [ 4178.401050] LustreError: 138741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.437974] LustreError: 138741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4183.239544] LustreError: 138722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313215 with bad export cookie 9278452807037520479 [ 4183.262284] LustreError: 138722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4183.264307] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4183.528133] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.547389] Lustre: Skipped 2 previous similar messages [ 4183.912559] Lustre: server umount lustre-MDT0001 complete [ 4189.711791] Lustre: server umount lustre-OST0000 complete [ 4201.416286] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 21:40:32 (1788313232) [ 4208.976078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.510208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.302336] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4214.766549] LustreError: 142430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4214.796927] LustreError: 142430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4223.288515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4228.318870] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4236.541705] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4241.980828] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4247.242619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4252.755254] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4254.717747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4260.931799] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4265.709210] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4265.902379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4270.046707] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4270.274742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4274.147585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4274.163066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.185955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.196674] Lustre: Skipped 1 previous similar message [ 4278.198953] Lustre: server umount lustre-OST0000 complete [ 4285.421542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4285.428945] LustreError: Skipped 1 previous similar message [ 4287.896445] Lustre: server umount lustre-MDT0000 complete [ 4290.531603] LustreError: 142430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.545477] LustreError: 142430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4291.304156] LustreError: 142410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313323 with bad export cookie 9278452807037521550 [ 4291.318245] LustreError: 142410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4291.332793] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4291.839059] Lustre: server umount lustre-MDT0001 complete [ 4300.642528] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 21:42:11 (1788313331) [ 4307.826469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4323.425111] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4328.543548] LustreError: 146589:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.102@tcp: failed processing log, type 4: rc = -110 [ 4354.272749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4354.277460] Lustre: Skipped 2 previous similar messages [ 4361.001232] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4368.956988] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4375.719287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4376.592961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4381.678522] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4391.462282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4397.294938] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4405.439739] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4411.145275] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4415.579764] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4415.777478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4421.252910] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4421.564234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4427.747992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4427.756280] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4427.767218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4427.767229] Lustre: Skipped 4 previous similar messages [ 4427.798896] Lustre: Skipped 4 previous similar messages [ 4430.407182] Lustre: server umount lustre-MDT0000 complete [ 4432.867250] LustreError: 148274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.883150] LustreError: 148274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4434.457948] LustreError: 149737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313466 with bad export cookie 9278452807037522572 [ 4434.463186] LustreError: 149737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4434.463691] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4434.980603] Lustre: server umount lustre-MDT0001 complete [ 4449.375547] Lustre: server umount lustre-OST0000 complete [ 4458.332850] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 21:44:49 (1788313489) [ 4465.511985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4481.183631] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4486.303625] LustreError: 150785:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.102@tcp: failed processing log, type 4: rc = -110 [ 4519.183604] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4528.329489] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4537.114832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.944741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4542.764532] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4553.984343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.643705] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4569.419691] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4575.779471] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4584.180814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.582841] Lustre: lustre-OST0001: new disk, initializing [ 4584.589918] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4584.601495] Lustre: 153659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4591.592728] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4593.834690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4593.845413] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4593.965681] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4602.070918] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4608.528098] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4608.846078] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4615.257932] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4615.636482] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4619.744270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4619.763080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4619.779223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4619.799091] Lustre: Skipped 2 previous similar messages [ 4619.815610] Lustre: Skipped 3 previous similar messages [ 4623.842895] Lustre: server umount lustre-OST0000 complete [ 4634.417204] Lustre: server umount lustre-OST0001 complete [ 4645.013264] Lustre: server umount lustre-MDT0000 complete [ 4649.136199] LustreError: 150792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313681 with bad export cookie 9278452807037523524 [ 4649.142476] LustreError: 150792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4649.144481] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4649.635870] Lustre: server umount lustre-MDT0001 complete [ 4675.584269] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 21:48:26 (1788313706) [ 4677.348450] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4679.108820] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4680.833287] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 21:48:31 (1788313711) [ 4688.111427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.452911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4688.488840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4688.802970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4688.816327] Lustre: Skipped 6 previous similar messages [ 4692.908762] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4703.115350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.212955] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4703.237695] Lustre: 157220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4708.650261] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4717.156571] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4723.945116] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4731.186300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.514802] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4739.185945] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4741.301535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4747.504929] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4752.522477] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4752.764540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4758.114644] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4758.533515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4761.571573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4761.582699] LustreError: Skipped 2 previous similar messages [ 4761.587203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4761.617685] Lustre: Skipped 4 previous similar messages [ 4761.633641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4761.643986] Lustre: Skipped 4 previous similar messages [ 4767.838921] Lustre: server umount lustre-OST0000 complete [ 4791.556385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.066957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4793.103100] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4797.245416] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4797.263029] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4799.779755] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4807.978923] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4813.586948] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4813.740491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4820.034196] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4820.394721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4826.867840] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff94fc87008000.ost_server_uuid 50 [ 4829.397910] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff94fc87008000.ost_server_uuid in FULL state after 0 sec [ 4829.594050] LustreError: 157227:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.2@tcp) returned error from blocking AST (req@000000003564fc43 x1875181906964736 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000060267c95/0x80c3afab2cb29f9e lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.2@tcp remote: 0x5c759f61d2ae9f52 expref: 5 pid: 157226 timeout: 4929 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4829.648981] LustreError: lustre-MDT0000: A client on nid 192.168.201.2@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4829.659604] LustreError: 157215:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.2@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000060267c95/0x80c3afab2cb29f9e lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.2@tcp remote: 0x5c759f61d2ae9f52 expref: 6 pid: 157226 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4834.275878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4834.290057] LustreError: Skipped 1 previous similar message [ 4835.523126] Lustre: server umount lustre-OST0000 complete [ 4854.751213] Lustre: 128719:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788313870/real 1788313870] req@0000000000b0c612 x1875181906968448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788313886 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4855.263255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4855.723846] Lustre: server umount lustre-MDT0000 complete [ 4859.218804] LustreError: 157210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788313891 with bad export cookie 9278452807037524707 [ 4859.221664] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4859.230157] LustreError: 157210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4865.890568] Lustre: server umount lustre-MDT0001 complete [ 4877.710033] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 4880.377313] Key type lgssc unregistered [ 4880.639899] LNet: 163341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4880.645459] LNetError: 163341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4880.660214] LNet: Removed LNI 192.168.201.102@tcp [ 4881.652946] Key type .llcrypt unregistered [ 4881.658272] Key type ._llcrypt unregistered [ 4912.811593] Key type ._llcrypt registered [ 4912.823555] Key type .llcrypt registered [ 4913.121183] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 4914.930582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4915.042528] alg: No test for adler32 (adler32-zlib) [ 4916.431933] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 4916.962654] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 4918.775193] Key type lgssc registered [ 4919.900304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4928.452803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4928.471329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4930.063088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4935.804842] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4940.269360] LustreError: 164760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4940.278440] LustreError: 164760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4945.380779] LustreError: 164761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4946.322210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4946.931746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4953.350621] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4962.331563] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4968.271250] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4975.171481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4975.565667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4976.589127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4982.737884] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4990.103951] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4996.994900] Lustre: Failing over lustre-MDT0000 [ 4997.091073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4997.094590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4997.100984] Lustre: Skipped 1 previous similar message [ 4997.226579] Lustre: server umount lustre-MDT0000 complete [ 4998.024571] LustreError: 164757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5002.215301] LustreError: 164755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5002.225578] LustreError: 164755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5007.331192] LustreError: 164761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5007.352774] LustreError: 164761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5009.413918] LustreError: 164742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.2@tcp arrived at 1788314041 with bad export cookie 18284379910402586881 [ 5009.443663] LustreError: 164742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5017.426388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5017.569232] LustreError: 164761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5017.598809] LustreError: 164761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5017.739658] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5018.118251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5018.216813] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5023.205418] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5023.224160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5023.774603] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5026.457621] Lustre: lustre-MDT0000: Denying connection for new client 226a1596-bea1-4c61-8d2c-eb3b33127409 (at 192.168.201.2@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5038.049659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5038.071360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.082912] Lustre: Skipped 1 previous similar message [ 5038.095269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5038.108119] Lustre: Skipped 2 previous similar messages [ 5043.719307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5043.741915] Lustre: Skipped 2 previous similar messages [ 5048.806710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5048.824654] Lustre: Skipped 1 previous similar message [ 5049.312460] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5049.697599] Lustre: server umount lustre-OST0000 complete [ 5054.208469] LustreError: 168496:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5054.217953] Lustre: 167843:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5054.225382] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5054.258752] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5054.270703] LustreError: 167843:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003f397672 x1875183087435264/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5054.313628] LustreError: 167843:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5054.326600] LustreError: 167843:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5054.347984] Lustre: lustre-MDT0000: Recovery over after 0:31, of 2 clients 0 recovered and 2 were evicted. [ 5054.974515] Lustre: server umount lustre-MDT0000 complete [ 5059.051101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5059.079140] Lustre: Skipped 1 previous similar message [ 5059.087779] LustreError: 164756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5060.037757] LustreError: 164740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314092 with bad export cookie 18284379910402586986 [ 5060.041933] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5060.047527] LustreError: 164071:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5060.048492] LustreError: 164740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5060.897689] Lustre: server umount lustre-MDT0001 complete [ 5076.171583] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 5081.558244] Key type lgssc unregistered [ 5082.063583] LNet: 169252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5082.076748] LNetError: 169252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5082.104429] LNet: Removed LNI 192.168.201.102@tcp [ 5083.215673] Key type .llcrypt unregistered [ 5083.217323] Key type ._llcrypt unregistered [ 5115.050399] Key type ._llcrypt registered [ 5115.053415] Key type .llcrypt registered [ 5115.201643] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5116.966335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5117.002961] alg: No test for adler32 (adler32-zlib) [ 5118.243158] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 5118.566225] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 5120.264054] Key type lgssc registered [ 5121.338345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5130.799789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5130.822863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5132.842160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5139.206096] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5142.000218] LustreError: 170666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5142.030264] LustreError: 170666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5147.117952] LustreError: 170667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5149.974983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5150.753903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5157.322506] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5165.248512] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5171.528862] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5179.013148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5179.390472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5186.962750] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5195.950537] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5202.943786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5213.152731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5213.161749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.170192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5213.196654] Lustre: Skipped 1 previous similar message [ 5218.286504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5218.294842] Lustre: Skipped 1 previous similar message [ 5219.209325] Lustre: server umount lustre-OST0000 complete [ 5227.489449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5227.497605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5227.511653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5227.514756] Lustre: Skipped 1 previous similar message [ 5228.514499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.640605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5233.644753] Lustre: Skipped 3 previous similar messages [ 5238.240523] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5238.752974] LustreError: 170666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.772343] LustreError: 170666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5240.488215] Lustre: server umount lustre-MDT0000 complete [ 5243.875390] LustreError: 171394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5243.889503] LustreError: 171394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5244.460520] LustreError: 170648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314276 with bad export cookie 3811055643941223439 [ 5244.467861] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5244.837280] Lustre: server umount lustre-MDT0001 complete [ 5258.253294] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 5261.593611] Key type lgssc unregistered [ 5261.940047] LNet: 174226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5261.950762] LNetError: 174226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5261.973023] LNet: Removed LNI 192.168.201.102@tcp [ 5263.061562] Key type .llcrypt unregistered [ 5263.066425] Key type ._llcrypt unregistered [ 5297.848530] Key type ._llcrypt registered [ 5297.850113] Key type .llcrypt registered [ 5298.047749] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5299.536826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5299.577845] alg: No test for adler32 (adler32-zlib) [ 5301.008383] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 5301.451228] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 5303.215243] Key type lgssc registered [ 5305.229636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5315.327899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5315.362134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5317.087369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.669829] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5327.335262] LustreError: 175927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5327.371060] LustreError: 175927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5332.454711] LustreError: 175928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5333.367723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.932437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5338.663609] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5347.279844] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5354.596625] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5363.068218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5363.534224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.576314] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5380.866287] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5387.086774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5397.473015] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5397.478082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5397.490206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5397.531823] Lustre: Skipped 1 previous similar message [ 5402.593148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5402.597303] Lustre: Skipped 1 previous similar message [ 5403.370720] Lustre: server umount lustre-OST0000 complete [ 5407.713366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5407.723196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5407.736659] Lustre: Skipped 1 previous similar message [ 5407.764957] Lustre: Skipped 1 previous similar message [ 5412.837592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5412.848629] Lustre: Skipped 1 previous similar message [ 5413.395202] Lustre: server umount lustre-MDT0000 complete [ 5417.267723] LustreError: 175908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314449 with bad export cookie 16722239226575962158 [ 5417.277355] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5417.615238] Lustre: server umount lustre-MDT0001 complete [ 5431.773203] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 5434.609431] Key type lgssc unregistered [ 5434.898320] LNet: 179499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5434.906403] LNetError: 179499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5434.925682] LNet: Removed LNI 192.168.201.102@tcp [ 5435.913454] Key type .llcrypt unregistered [ 5435.915903] Key type ._llcrypt unregistered [ 5458.133394] Key type ._llcrypt registered [ 5458.134729] Key type .llcrypt registered [ 5458.217989] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5459.480904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5459.500237] alg: No test for adler32 (adler32-zlib) [ 5460.503516] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 5460.629976] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 5462.271212] Key type lgssc registered [ 5463.732307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5474.082299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5474.113588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5475.551416] Lustre: *** cfs_fail_loc=135, val=0*** [ 5475.646248] LustreError: 181022:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5475.653720] LustreError: 181022:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.102@tcp: cfg command failed: rc = -2 [ 5475.665336] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5475.665336] [ 5475.673699] LustreError: MGC192.168.201.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5475.688372] LustreError: 180982:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5475.698101] LustreError: 180982:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5475.706062] LustreError: 180982:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5475.716143] LustreError: 180982:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5475.825656] Lustre: server umount lustre-MDT0000 complete [ 5475.829247] LustreError: 180982:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5482.528689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.894047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5487.318530] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5492.195621] LustreError: 181392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.209964] LustreError: 181392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5495.674936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5495.941389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5500.307650] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5507.183344] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5512.789681] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5516.259969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5516.269190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5516.280698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5517.795824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5517.797753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5520.936058] LustreError: 181393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5521.095830] Lustre: server umount lustre-MDT0000 complete [ 5523.808980] LustreError: 181373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314555 with bad export cookie 4125548637800514888 [ 5523.819096] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5523.823651] LustreError: 181373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5524.046816] Lustre: server umount lustre-MDT0001 complete [ 5534.067699] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 5536.924476] Key type lgssc unregistered [ 5537.141290] LNet: 183716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5537.146899] LNetError: 183716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5537.160346] LNet: Removed LNI 192.168.201.102@tcp [ 5537.827314] Key type .llcrypt unregistered [ 5537.830336] Key type ._llcrypt unregistered [ 5560.487679] Key type ._llcrypt registered [ 5560.490249] Key type .llcrypt registered [ 5560.588697] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 5560.839358] Key type .llcrypt unregistered [ 5560.841979] Key type ._llcrypt unregistered [ 5577.690282] Key type ._llcrypt registered [ 5577.692582] Key type .llcrypt registered [ 5577.791807] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5579.455060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5579.579835] alg: No test for adler32 (adler32-zlib) [ 5580.790942] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 5581.060807] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 5582.775214] Key type lgssc registered [ 5583.717641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5590.523929] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5590.533914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5605.983664] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5611.103376] LustreError: 185852:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.102@tcp: failed processing log, type 4: rc = -110 [ 5637.791354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5643.717887] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5650.331939] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5657.131902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5657.736786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5657.798949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5662.087155] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5663.206803] LustreError: 185878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5663.222069] LustreError: 185878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5668.332894] LustreError: 185877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5670.616879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.005945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5675.421504] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5681.541566] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5687.592697] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5690.324337] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5699.043509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5699.053434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.082926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5701.602623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5701.610206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.627599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5703.148355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5703.150831] Lustre: Skipped 1 previous similar message [ 5708.259205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5708.277349] Lustre: Skipped 1 previous similar message [ 5713.375253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5713.381399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5713.413394] Lustre: Skipped 1 previous similar message [ 5713.521146] Lustre: server umount lustre-OST0000 complete [ 5718.504724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.519137] Lustre: Skipped 1 previous similar message [ 5723.617506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5723.626015] Lustre: Skipped 3 previous similar messages [ 5732.319140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5732.502991] Lustre: server umount lustre-MDT0000 complete [ 5733.867517] LustreError: 186890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5733.880461] LustreError: 186890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5736.437978] LustreError: 188514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314768 with bad export cookie 8600815631884172963 [ 5736.442380] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5736.447085] LustreError: 188514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5736.811901] Lustre: server umount lustre-MDT0001 complete [ 5746.923339] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 5749.659661] Key type lgssc unregistered [ 5749.894760] LNet: 189774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5749.899396] LNetError: 189774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5750.950551] LNet: Removed LNI 192.168.201.102@tcp [ 5751.612771] Key type .llcrypt unregistered [ 5751.631085] Key type ._llcrypt unregistered [ 5781.194876] Key type ._llcrypt registered [ 5781.197490] Key type .llcrypt registered [ 5781.339401] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5782.412795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5782.547218] alg: No test for adler32 (adler32-zlib) [ 5783.682207] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 5783.968949] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 5785.722675] Key type lgssc registered [ 5787.303480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5795.055859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5795.072505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5796.523853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5801.369653] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5806.054746] LustreError: 191188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5806.089951] LustreError: 191188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5810.112489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5810.508108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5815.035320] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5823.213185] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5829.230181] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5835.597178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5836.059834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5842.237635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5842.493733] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5848.975345] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5853.648906] Lustre: Failing over lustre-MDT0000 [ 5853.853903] Lustre: server umount lustre-MDT0000 complete [ 5855.712627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.717263] LustreError: 191915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.726121] Lustre: Skipped 1 previous similar message [ 5860.833883] LustreError: 192429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5860.854881] LustreError: 192429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5865.954581] LustreError: 191187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.977321] LustreError: 191187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5871.071242] Lustre: 190500:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788314887/real 1788314887] req@00000000a94686b6 x1875183996527232/t0(0) o400->MGC192.168.201.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788314903 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5871.075656] LustreError: 191915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.110816] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5871.139526] LustreError: 191915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5871.812370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5881.313227] LustreError: 191915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5881.315128] LustreError: 190498:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ecbc6262 x1875183996534272/t0(0) o250->MGC192.168.201.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5881.323684] LustreError: 191915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5881.739112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5881.801636] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5883.239977] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5886.964472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5887.006812] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5887.078681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5887.396193] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5892.241345] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5899.473978] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5912.549854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5912.552074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5912.563551] Lustre: Skipped 2 previous similar messages [ 5912.581102] Lustre: Skipped 1 previous similar message [ 5916.039139] Lustre: server umount lustre-OST0000 complete [ 5922.784330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5922.787265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5922.788547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5922.806026] Lustre: Skipped 1 previous similar message [ 5922.810539] Lustre: Skipped 1 previous similar message [ 5926.983426] Lustre: server umount lustre-MDT0000 complete [ 5927.904712] LustreError: 191188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5927.929270] LustreError: 191188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5931.532080] LustreError: 191168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788314963 with bad export cookie 6864286136925274216 [ 5931.534703] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5931.550090] LustreError: 191168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5932.001581] Lustre: server umount lustre-MDT0001 complete [ 5941.701086] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 5944.107971] Key type lgssc unregistered [ 5944.359752] LNet: 196085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5944.369593] LNetError: 196085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5944.381120] LNet: Removed LNI 192.168.201.102@tcp [ 5945.164270] Key type .llcrypt unregistered [ 5945.166320] Key type ._llcrypt unregistered [ 5969.529305] Key type ._llcrypt registered [ 5969.531350] Key type .llcrypt registered [ 5969.619043] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5970.825836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5970.881810] alg: No test for adler32 (adler32-zlib) [ 5971.953954] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 5972.165486] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 5973.847392] Key type lgssc registered [ 5974.879900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5982.852807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5982.876976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5984.441391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5988.892146] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5993.968367] LustreError: 197479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.991284] LustreError: 197479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5997.765220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.348298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.011730] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6010.872194] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6015.690727] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6022.498872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6022.917147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6028.506682] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6029.132373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6036.554368] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6040.385294] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6048.036476] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6060.549903] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6070.241329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.248222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6070.268365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.459731] Lustre: server umount lustre-OST0000 complete [ 6080.516352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.520940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6080.528246] Lustre: Skipped 1 previous similar message [ 6080.530835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6080.530843] Lustre: Skipped 1 previous similar message [ 6083.720405] Lustre: server umount lustre-MDT0000 complete [ 6085.600699] LustreError: 197475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6085.637896] LustreError: 197475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6088.675505] LustreError: 197458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315120 with bad export cookie 12656833755628620631 [ 6088.692433] LustreError: 197458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6088.693474] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6089.262438] Lustre: server umount lustre-MDT0001 complete [ 6103.206196] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 6105.836806] Key type lgssc unregistered [ 6106.155248] LNet: 201332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6106.160790] LNetError: 201332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6107.181789] LNet: Removed LNI 192.168.201.102@tcp [ 6108.200633] Key type .llcrypt unregistered [ 6108.202860] Key type ._llcrypt unregistered [ 6129.549598] Key type ._llcrypt registered [ 6129.551780] Key type .llcrypt registered [ 6129.639151] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 6131.083240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6131.126054] alg: No test for adler32 (adler32-zlib) [ 6132.214523] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 6132.391279] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 6134.047244] Key type lgssc registered [ 6135.417920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6144.239681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6144.257266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.863283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.046424] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6155.238964] LustreError: 202600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.255792] LustreError: 202600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6160.058300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6160.467818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6165.366250] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6173.869636] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6179.550757] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6186.494513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6186.873296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.951670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6192.390692] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6200.644985] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6207.717900] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6217.630750] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6226.516567] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6236.638859] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6246.368283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6246.370473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.371422] LustreError: Skipped 1 previous similar message [ 6246.372325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6246.376029] Lustre: Skipped 1 previous similar message [ 6250.253129] Lustre: server umount lustre-OST0000 complete [ 6256.609257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.611209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6256.632174] Lustre: Skipped 1 previous similar message [ 6256.634525] Lustre: Skipped 1 previous similar message [ 6260.498676] Lustre: server umount lustre-MDT0000 complete [ 6261.730388] LustreError: 203328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6261.752521] LustreError: 203328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6265.026552] LustreError: 205628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315297 with bad export cookie 11772346257860204579 [ 6265.040231] LustreError: 205628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6265.040394] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6265.446245] Lustre: server umount lustre-MDT0001 complete [ 6277.165679] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 6280.000982] Key type lgssc unregistered [ 6280.352837] LNet: 207527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6280.356446] LNetError: 207527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6281.383325] LNet: Removed LNI 192.168.201.102@tcp [ 6282.221528] Key type .llcrypt unregistered [ 6282.225134] Key type ._llcrypt unregistered [ 6305.799923] Key type ._llcrypt registered [ 6305.801840] Key type .llcrypt registered [ 6305.902915] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 6307.276091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6307.295379] alg: No test for adler32 (adler32-zlib) [ 6308.373715] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 6308.642190] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 6310.303348] Key type lgssc registered [ 6311.501554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6320.263782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6320.301646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6321.760612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6326.043387] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6331.874754] LustreError: 208941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.882846] LustreError: 208941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6334.722792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.970656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6339.545376] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6346.963342] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6351.901870] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6358.753853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6359.077294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6364.068420] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6366.272861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6372.713465] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6390.761233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6390.771356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.787098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6391.784737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.787990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.789798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6395.311377] Lustre: server umount lustre-OST0000 complete [ 6401.504758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6401.510407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.518641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6401.524489] Lustre: Skipped 1 previous similar message [ 6405.718545] Lustre: server umount lustre-MDT0000 complete [ 6407.136691] LustreError: 208941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6407.154721] LustreError: 208941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6408.950961] LustreError: 209675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315441 with bad export cookie 15526094824463141966 [ 6408.958261] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6408.964962] LustreError: 209675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6409.386850] Lustre: server umount lustre-MDT0001 complete [ 6421.498919] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 6424.750567] Key type lgssc unregistered [ 6425.066875] LNet: 212625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6425.090756] LNetError: 212625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6425.126176] LNet: Removed LNI 192.168.201.102@tcp [ 6426.194295] Key type .llcrypt unregistered [ 6426.195772] Key type ._llcrypt unregistered [ 6446.289794] Key type ._llcrypt registered [ 6446.291738] Key type .llcrypt registered [ 6446.409924] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 6447.504704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6447.626374] alg: No test for adler32 (adler32-zlib) [ 6448.814115] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 6449.030093] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 6450.735177] Key type lgssc registered [ 6452.022666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6460.810418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6460.823213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6462.422392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6468.541814] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6472.688276] LustreError: 213894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6472.719647] LustreError: 213894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6477.801162] LustreError: 213893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6478.116400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.621254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6483.954633] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6492.595953] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6499.307825] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6507.362969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6507.751880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6515.134052] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6522.320814] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6532.364251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6553.061427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6553.083251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6553.089646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6553.106245] Lustre: Skipped 1 previous similar message [ 6555.636444] Lustre: server umount lustre-OST0000 complete [ 6560.744068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6560.757359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6560.783661] Lustre: Skipped 1 previous similar message [ 6560.792462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6563.297480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6563.306360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6563.316482] Lustre: Skipped 1 previous similar message [ 6566.122926] Lustre: server umount lustre-MDT0000 complete [ 6568.416286] LustreError: 213889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6568.430548] LustreError: 213889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6570.308551] LustreError: 213874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315602 with bad export cookie 10878406294666097796 [ 6570.322092] LustreError: 213874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6570.323719] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6570.772037] Lustre: server umount lustre-MDT0001 complete [ 6582.284276] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 6585.126285] Key type lgssc unregistered [ 6585.400272] LNet: 217604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6585.408849] LNetError: 217604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6585.420963] LNet: Removed LNI 192.168.201.102@tcp [ 6586.293461] Key type .llcrypt unregistered [ 6586.295643] Key type ._llcrypt unregistered [ 6605.039102] Key type ._llcrypt registered [ 6605.043862] Key type .llcrypt registered [ 6605.206253] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 6606.056566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6606.147913] alg: No test for adler32 (adler32-zlib) [ 6607.451346] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 6607.739163] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 6609.415169] Key type lgssc registered [ 6610.872490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6618.854081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6618.875583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.375204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6625.408051] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6625.767532] LustreError: 218870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.785581] LustreError: 218870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6630.881060] LustreError: 218871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6634.090988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.521941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6638.549499] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6646.154796] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6651.598560] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6658.553880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6658.902491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6665.079302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6665.730981] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6673.628409] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6682.993884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6683.148080] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6683.380535] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6689.032361] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6696.554456] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6698.300326] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6710.398435] Lustre: Permanently deactivating lustre-OST0001 [ 6710.412482] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6719.037847] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6724.929879] Lustre: Permanently reactivating lustre-OST0001 [ 6724.940788] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6724.946236] Lustre: Skipped 2 previous similar messages [ 6728.411536] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6728.428180] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6728.433452] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6728.448179] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6728.517866] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6737.388429] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6737.396767] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6737.399652] Lustre: Skipped 2 previous similar messages [ 6737.970129] Lustre: server umount lustre-OST0001 complete [ 6742.499890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6742.517074] LustreError: 220691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6742.519807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6742.541336] LustreError: 220691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6742.556881] Lustre: Skipped 2 previous similar messages [ 6747.617765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6747.620870] LustreError: 220691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6747.630740] Lustre: Skipped 1 previous similar message [ 6747.647776] LustreError: 220691:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6748.435500] Lustre: server umount lustre-OST0000 complete [ 6752.224777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6752.230065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.240300] Lustre: Skipped 1 previous similar message [ 6752.245087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6757.858038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6757.876252] Lustre: Skipped 2 previous similar messages [ 6758.473445] Lustre: server umount lustre-MDT0000 complete [ 6762.609978] LustreError: 218850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788315794 with bad export cookie 9850886648396185065 [ 6762.612532] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6762.624977] LustreError: 218850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6762.916632] Lustre: server umount lustre-MDT0001 complete [ 6773.459289] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 6776.111745] Key type lgssc unregistered [ 6776.401120] LNet: 223920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6776.407358] LNetError: 223920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6776.417635] LNet: Removed LNI 192.168.201.102@tcp [ 6777.309635] Key type .llcrypt unregistered [ 6777.312027] Key type ._llcrypt unregistered [ 6818.251966] Key type ._llcrypt registered [ 6818.255533] Key type .llcrypt registered [ 6818.311536] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 6819.726212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6819.768827] alg: No test for adler32 (adler32-zlib) [ 6820.978793] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 6821.273805] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 6823.007804] Key type lgssc registered [ 6824.692360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6834.383552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6834.418742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.656309] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6835.696657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6835.838993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6840.864955] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6848.975824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6849.041229] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6849.065922] Lustre: 226123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6849.340739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6853.182126] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6860.715598] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6866.353515] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6872.725775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6872.969594] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6873.188722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6880.903471] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6882.594515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6889.081619] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6891.933805] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6897.995634] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6904.797274] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6911.389249] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6925.221759] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6925.229167] Lustre: Skipped 1 previous similar message [ 6944.967648] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6944.983284] Lustre: Skipped 2 previous similar messages [ 6983.797943] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6983.808567] Lustre: Skipped 4 previous similar messages [ 7051.755725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7051.758408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7051.761692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7051.770865] Lustre: Skipped 1 previous similar message [ 7051.774364] LustreError: Skipped 1 previous similar message [ 7055.020630] Lustre: server umount lustre-OST0000 complete [ 7059.441950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7059.461939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7059.473756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7059.477862] Lustre: Skipped 1 previous similar message [ 7061.991628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7061.995808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7067.107895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7067.122534] Lustre: Skipped 2 previous similar messages [ 7071.201617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7071.216964] Lustre: Skipped 1 previous similar message [ 7073.251327] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7073.626854] Lustre: server umount lustre-MDT0000 complete [ 7078.745926] LustreError: 228690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316110 with bad export cookie 15396272715790728023 [ 7078.750746] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7078.760145] LustreError: 228690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7079.420737] Lustre: server umount lustre-MDT0001 complete [ 7092.505824] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 7095.893736] Key type lgssc unregistered [ 7096.180570] LNet: 230927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7096.184949] LNetError: 230927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7097.253784] LNet: Removed LNI 192.168.201.102@tcp [ 7097.969870] Key type .llcrypt unregistered [ 7097.975196] Key type ._llcrypt unregistered [ 7121.785536] Key type ._llcrypt registered [ 7121.787466] Key type .llcrypt registered [ 7121.879572] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7123.179677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7123.263093] alg: No test for adler32 (adler32-zlib) [ 7124.358612] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 7124.792457] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 7126.559191] Key type lgssc registered [ 7128.317163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7137.822557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7137.849244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7139.700625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7145.512855] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7149.098067] LustreError: 232346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7149.105653] LustreError: 232346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7154.153272] LustreError: 232345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.769785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7158.183500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7164.198630] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7172.186521] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7177.902906] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7184.898322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7185.447092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7192.460954] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7199.437640] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7208.951944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7229.428165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7229.430304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.457620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7229.472644] Lustre: Skipped 1 previous similar message [ 7234.528440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7239.657129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7239.665825] Lustre: Skipped 2 previous similar messages [ 7242.207150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7242.422263] Lustre: server umount lustre-OST0000 complete [ 7249.888737] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7249.904188] Lustre: Skipped 1 previous similar message [ 7249.908268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7252.857604] Lustre: server umount lustre-MDT0000 complete [ 7255.013258] LustreError: 232346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.030264] LustreError: 232346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7256.881646] LustreError: 232325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788316288 with bad export cookie 10454151294541547086 [ 7256.889300] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7256.897402] LustreError: 232325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7257.453522] Lustre: server umount lustre-MDT0001 complete [ 7270.163261] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 7272.630449] Key type lgssc unregistered [ 7272.941927] LNet: 236061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7272.949283] LNetError: 236061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7272.975251] LNet: Removed LNI 192.168.201.102@tcp [ 7274.006655] Key type .llcrypt unregistered [ 7274.008370] Key type ._llcrypt unregistered [ 7298.869722] Key type ._llcrypt registered [ 7298.876121] Key type .llcrypt registered [ 7299.053829] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7300.381650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7300.422780] alg: No test for adler32 (adler32-zlib) [ 7301.580670] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 7301.976420] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 7303.679167] Key type lgssc registered [ 7305.061044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7578.559873] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 7582.464891] Key type lgssc unregistered [ 7582.882845] LNet: 238086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7582.897838] LNetError: 238086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7582.922358] LNet: Removed LNI 192.168.201.102@tcp [ 7584.148355] Key type .llcrypt unregistered [ 7584.161890] Key type ._llcrypt unregistered [ 7619.395691] Key type ._llcrypt registered [ 7619.398832] Key type .llcrypt registered [ 7619.522519] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 7637.941869] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7639.251374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7639.296848] alg: No test for adler32 (adler32-zlib) [ 7640.415294] Lustre: Lustre: Build Version: 2.17.57_45_g07d9be3 [ 7640.882339] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 7642.591166] Key type lgssc registered [ 7644.043536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7679.412689] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7439 sec ========== 22:38:30 (1788316710) [ 7681.313548] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:38:32 (1788316712) === [ 7686.181806] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:38:36 (1788316716) === [ 7730.968360] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 7734.515820] Key type lgssc unregistered [ 7734.939387] LNet: 243334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7734.950083] LNetError: 243334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7734.961843] LNet: Removed LNI 192.168.201.102@tcp [ 7735.872811] Key type .llcrypt unregistered [ 7735.877342] Key type ._llcrypt unregistered