[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450792212 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003252] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.006026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.007010] pid_max: default: 32768 minimum: 301 [ 0.009025] LSM: Security Framework initializing [ 0.010041] Yama: becoming mindful. [ 0.011030] SELinux: Initializing. [ 0.012055] *** VALIDATE selinux *** [ 0.020240] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024085] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025121] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026063] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027095] *** VALIDATE tmpfs *** [ 0.029036] *** VALIDATE proc *** [ 0.029962] *** VALIDATE cgroup *** [ 0.030006] *** VALIDATE cgroup2 *** [ 0.031090] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032122] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034022] Spectre V2 : User space: Vulnerable [ 0.035005] Speculative Store Bypass: Vulnerable [ 0.038284] debug: unmapping init [mem 0xffffffffa3859000-0xffffffffa3860fff] [ 0.040000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040635] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041014] ... version: 2 [ 0.042008] ... bit width: 48 [ 0.043006] ... generic registers: 4 [ 0.044007] ... value mask: 0000ffffffffffff [ 0.045009] ... max period: 00007fffffffffff [ 0.046007] ... fixed-purpose events: 3 [ 0.046887] ... event mask: 000000070000000f [ 0.048216] rcu: Hierarchical SRCU implementation. [ 0.050147] smp: Bringing up secondary CPUs ... [ 0.051526] x86: Booting SMP configuration: [ 0.052025] .... node #0, CPUs: #1 #2 #3 [ 0.055557] smp: Brought up 1 node, 4 CPUs [ 0.057008] smpboot: Max logical packages: 1 [ 0.058012] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.203200] node 0 deferred pages initialised in 143ms [ 0.206138] devtmpfs: initialized [ 0.207201] x86/mm: Memory block size: 128MB [ 0.209671] gcov: version magic: 0x41383552 [ 0.211084] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.212074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.213240] pinctrl core: initialized pinctrl subsystem [ 0.214168] [ 0.214624] ************************************************************* [ 0.215009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.216008] ** ** [ 0.217008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.218010] ** ** [ 0.219006] ** This means that this kernel is built to expose internal ** [ 0.220009] ** IOMMU data structures, which may compromise security on ** [ 0.221009] ** your system. ** [ 0.222005] ** ** [ 0.223008] ** If you see this message and you are not debugging the ** [ 0.224009] ** kernel, report this immediately to your vendor! ** [ 0.225009] ** ** [ 0.226007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.227007] ************************************************************* [ 0.228808] NET: Registered protocol family 16 [ 0.229470] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.230040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.231045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.232560] cpuidle: using governor menu [ 0.234330] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.238073] PCI: Using configuration type 1 for base access [ 0.240111] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.254042] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.255027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.256193] cryptd: max_cpu_qlen set to 1000 [ 0.259016] ACPI: Added _OSI(Module Device) [ 0.260009] ACPI: Added _OSI(Processor Device) [ 0.261010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.263009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.266941] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.273546] ACPI: Interpreter enabled [ 0.275045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.276018] ACPI: Using IOAPIC for interrupt routing [ 0.277071] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.280343] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.287799] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.290029] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.292013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.294067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.297990] acpiphp: Slot [2] registered [ 0.298082] acpiphp: Slot [5] registered [ 0.299092] acpiphp: Slot [6] registered [ 0.299965] acpiphp: Slot [7] registered [ 0.301047] acpiphp: Slot [8] registered [ 0.301956] acpiphp: Slot [9] registered [ 0.303048] acpiphp: Slot [10] registered [ 0.304022] acpiphp: Slot [3] registered [ 0.304877] acpiphp: Slot [4] registered [ 0.306214] acpiphp: Slot [11] registered [ 0.307138] acpiphp: Slot [12] registered [ 0.308066] acpiphp: Slot [13] registered [ 0.309063] acpiphp: Slot [14] registered [ 0.310058] acpiphp: Slot [15] registered [ 0.312063] acpiphp: Slot [16] registered [ 0.313086] acpiphp: Slot [17] registered [ 0.314061] acpiphp: Slot [18] registered [ 0.315068] acpiphp: Slot [19] registered [ 0.316063] acpiphp: Slot [20] registered [ 0.317075] acpiphp: Slot [21] registered [ 0.318050] acpiphp: Slot [22] registered [ 0.319065] acpiphp: Slot [23] registered [ 0.320060] acpiphp: Slot [24] registered [ 0.321055] acpiphp: Slot [25] registered [ 0.322045] acpiphp: Slot [26] registered [ 0.323068] acpiphp: Slot [27] registered [ 0.324066] acpiphp: Slot [28] registered [ 0.325046] acpiphp: Slot [29] registered [ 0.326091] acpiphp: Slot [30] registered [ 0.327074] acpiphp: Slot [31] registered [ 0.328155] PCI host bridge to bus 0000:00 [ 0.329012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.331013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.335034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.337023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.340181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342971] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.344833] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.357054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.360019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.362010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364439] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.366565] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.368026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.370582] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.374014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.384000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.388018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392166] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.397012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.401011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.418016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.427428] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.443020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.450015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.469021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.485597] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.494020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.500015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.515019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.525622] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.536014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.541017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.553017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.565414] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.575015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.583020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.598030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.610121] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.618013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.624014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.639015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.650871] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.654381] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.656298] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.659294] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.661188] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.666661] iommu: Default domain type: Passthrough [ 0.668391] SCSI subsystem initialized [ 0.669101] ACPI: bus type USB registered [ 0.670080] usbcore: registered new interface driver usbfs [ 0.672048] usbcore: registered new interface driver hub [ 0.674070] usbcore: registered new device driver usb [ 0.675138] pps_core: LinuxPPS API ver. 1 registered [ 0.677007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.680047] PTP clock support registered [ 0.682152] EDAC MC: Ver: 3.0.0 [ 0.684183] PCI: Using ACPI for IRQ routing [ 0.685986] NetLabel: Initializing [ 0.687007] NetLabel: domain hash size = 128 [ 0.689006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.690094] NetLabel: unlabeled traffic allowed by default [ 0.692147] vgaarb: loaded [ 0.694684] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.696011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.702104] clocksource: Switched to clocksource kvm-clock [ 0.816285] VFS: Disk quotas dquot_6.6.0 [ 0.817504] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.819412] *** VALIDATE ramfs *** [ 0.820487] *** VALIDATE hugetlbfs *** [ 0.822326] pnp: PnP ACPI init [ 0.824333] pnp: PnP ACPI: found 6 devices [ 0.845840] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.848416] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.850055] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.851621] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.853381] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.855358] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.857982] NET: Registered protocol family 2 [ 0.859887] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.864177] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.866878] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.870980] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.873584] TCP: Hash tables configured (established 65536 bind 65536) [ 0.875720] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.877920] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880035] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.882150] NET: Registered protocol family 1 [ 0.884625] RPC: Registered named UNIX socket transport module. [ 0.886643] RPC: Registered udp transport module. [ 0.887797] RPC: Registered tcp transport module. [ 0.888786] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.890323] NET: Registered protocol family 44 [ 0.891462] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.892960] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.894353] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.896509] PCI: CLS 0 bytes, default 64 [ 0.897772] Unpacking initramfs... [ 2.214861] debug: unmapping init [mem 0xffff8e607cc54000-0xffff8e607ffbffff] [ 2.220559] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.222168] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.224523] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.700521] Initialise system trusted keyrings [ 2.701739] Key type blacklist registered [ 2.703212] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.711109] zbud: loaded [ 2.714382] *** VALIDATE nfs *** [ 2.715415] *** VALIDATE nfs4 *** [ 2.716759] pstore: using deflate compression [ 2.719821] Platform Keyring initialized [ 2.812410] NET: Registered protocol family 38 [ 2.813849] Key type asymmetric registered [ 2.815142] Asymmetric key parser 'x509' registered [ 2.816769] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.819286] io scheduler mq-deadline registered [ 2.820638] io scheduler kyber registered [ 2.821935] io scheduler bfq registered [ 2.826447] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.828891] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.831316] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.836431] ACPI: Power Button [PWRF] [ 2.843776] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.849976] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.862772] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.869660] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.883770] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.917137] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.946123] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.950592] Non-volatile memory driver v1.3 [ 2.951658] Linux agpgart interface v0.103 [ 2.982726] virtio_blk virtio1: [vda] 145248 512-byte logical blocks (74.4 MB/70.9 MiB) [ 2.984971] vda: detected capacity change from 0 to 74366976 [ 3.001806] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.004332] vdb: detected capacity change from 0 to 1073741824 [ 3.030271] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.032413] vdc: detected capacity change from 0 to 2621440000 [ 3.050846] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.053456] vdd: detected capacity change from 0 to 2621440000 [ 3.069695] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.071836] vde: detected capacity change from 0 to 4294967296 [ 3.109112] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.111223] vdf: detected capacity change from 0 to 4294967296 [ 3.131587] libphy: Fixed MDIO Bus: probed [ 3.170588] usbcore: registered new interface driver usbserial_generic [ 3.172315] usbserial: USB Serial support registered for generic [ 3.173891] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.177413] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.178613] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.181685] mousedev: PS/2 mouse device common for all mice [ 3.184502] rtc_cmos 00:05: RTC can wake from S4 [ 3.186374] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.188190] rtc_cmos 00:05: registered as rtc0 [ 3.192245] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.194344] intel_pstate: CPU model not supported [ 3.196979] hid: raw HID events driver (C) Jiri Kosina [ 3.198842] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.203025] usbcore: registered new interface driver usbhid [ 3.209868] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.213280] usbhid: USB HID core driver [ 3.213471] drop_monitor: Initializing network drop monitor service [ 3.213612] Initializing XFRM netlink socket [ 3.213971] NET: Registered protocol family 10 [ 3.215277] Segment Routing with IPv6 [ 3.222982] NET: Registered protocol family 17 [ 3.224391] mpls_gso: MPLS GSO support [ 3.228802] RAS: Correctable Errors collector initialized. [ 3.230546] AVX version of gcm_enc/dec engaged. [ 3.231904] AES CTR mode by8 optimization enabled [ 3.292170] sched_clock: Marking stable (3292121973, 0)->(4071458245, -779336272) [ 3.296450] registered taskstats version 1 [ 3.298346] Loading compiled-in X.509 certificates [ 3.299710] zswap: loaded using pool lzo/zbud [ 3.318325] Key type big_key registered [ 3.328371] Key type encrypted registered [ 3.329904] ima: No TPM chip found, activating TPM-bypass! [ 3.331827] ima: Allocated hash algorithm: sha1 [ 3.333443] ima: No architecture policies found [ 3.334952] evm: Initialising EVM extended attributes: [ 3.336387] evm: security.selinux [ 3.337201] evm: security.ima [ 3.337880] evm: security.capability [ 3.338737] evm: HMAC attrs: 0x1 [ 3.340641] rtc_cmos 00:05: setting system clock to 2026-06-29 17:57:48 UTC (1782755868) [ 3.345752] debug: unmapping init [mem 0xffffffffa4803000-0xffffffffa49fffff] [ 3.348343] debug: unmapping init [mem 0xffffffffa3582000-0xffffffffa3858fff] [ 3.352055] Write protecting the kernel read-only data: 28672k [ 3.354676] debug: unmapping init [mem 0xffffffffa1c03000-0xffffffffa1dfffff] [ 3.356634] debug: unmapping init [mem 0xffffffffa2514000-0xffffffffa25fffff] [ 3.384446] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.390535] systemd[1]: Detected virtualization kvm. [ 3.391776] systemd[1]: Detected architecture x86-64. [ 3.392997] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.417832] systemd[1]: No hostname configured. [ 3.418926] systemd[1]: Set hostname to . [ 3.420275] random: systemd: uninitialized urandom read (16 bytes read) [ 3.421959] systemd[1]: Initializing machine ID from random generator. [ 3.573526] random: systemd: uninitialized urandom read (16 bytes read) [ 3.575305] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.578238] random: systemd: uninitialized urandom read (16 bytes read) [ 3.580162] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.583258] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.310946] device-mapper: uevent: version 1.0.3 [ 4.312750] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.065610] virtio_net virtio0 ens2: renamed from eth0 [ 5.103944] random: fast init done [ 5.365936] scsi host0: ata_piix [ 5.404305] scsi host1: ata_piix [ 5.405718] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.411035] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.472713] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.843406] random: crng init done [ 9.844692] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.856890] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.078687] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.291447] SELinux: Disabled at runtime. [ 12.351835] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.359097] systemd[1]: Detected virtualization kvm. [ 12.360208] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.782529] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.785510] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.797157] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.800866] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.803884] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.811841] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.821743] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ 13.037939] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.409597] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.739652] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.827211] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.983325] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.110738] EDAC sbridge: Ver: 1.1.2 [ 15.812455] Key type dns_resolver registered [ 16.166987] NFS: Registering the id_resolver key type [ 16.168628] Key type id_resolver registered [ 16.169965] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg228-server login: [ 39.165069] libcfs: loading out-of-tree module taints kernel. [ 39.181590] Key type ._llcrypt registered [ 39.182459] Key type .llcrypt registered [ 39.217149] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 47.613244] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 48.238759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.245845] alg: No test for adler32 (adler32-zlib) [ 49.324657] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 49.633236] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 51.255392] Key type lgssc registered [ 51.915581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 59.956414] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 92.284708] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 103.180346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 103.233776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 104.702828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 104.754844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 104.914835] Lustre: lustre-MDT0000: new disk, initializing [ 105.077972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 105.110682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 111.562841] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.184454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 132.397595] Lustre: 6484:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 132.433242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 132.440932] Lustre: Skipped 1 previous similar message [ 132.618816] Lustre: lustre-MDT0001: new disk, initializing [ 132.759965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 132.807772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 132.819068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 139.720793] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 145.442228] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 157.309887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 157.703083] Lustre: lustre-OST0000: new disk, initializing [ 157.725519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 157.740307] Lustre: 8423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 157.868750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.857232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 163.862507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 163.912686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 164.243711] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.301237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.412429] Lustre: lustre-OST0001: new disk, initializing [ 176.417558] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 176.424169] Lustre: 9493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 176.488775] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.891139] hrtimer: interrupt took 9040772 ns [ 182.024272] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.335641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 182.353148] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 182.430799] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 191.603950] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 196.372788] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 201.254943] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing check_logdir /tmp/testlogs/ [ 205.612376] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing yml_node [ 208.942188] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 211.083216] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 212.995469] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 214.146639] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 29 14:01:17 EDT 2026 [ 227.530321] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 228.966888] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 244.192231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 244.192754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 244.206211] Lustre: Skipped 1 previous similar message [ 249.317427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 249.333385] Lustre: Skipped 6 previous similar messages [ 252.143177] Lustre: server umount lustre-MDT0000 complete [ 255.799344] LustreError: 10365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782756120 with bad export cookie 16079032352655498686 [ 255.805576] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 255.811954] LustreError: 10365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 259.557051] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.566823] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 259.566824] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 259.566839] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 259.571904] Lustre: Skipped 3 previous similar messages [ 259.609169] Lustre: Skipped 1 previous similar message [ 260.580754] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 260.602757] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 265.696358] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.696366] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 265.696371] Lustre: Skipped 2 previous similar messages [ 265.715192] LustreError: 6491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 270.305276] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 270.524576] Lustre: server umount lustre-MDT0001 complete [ 273.998356] Lustre: server umount lustre-OST0000 complete [ 277.590480] Lustre: server umount lustre-OST0001 complete [ 283.022961] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 291.825371] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 324.437502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 324.572591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 324.591727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 324.645993] Lustre: lustre-MDT0000: new disk, initializing [ 324.696313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 324.725464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 328.656802] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 338.245735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.382650] Lustre: 15056:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 338.457332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 338.463479] Lustre: Skipped 1 previous similar message [ 338.556142] Lustre: lustre-MDT0001: new disk, initializing [ 338.635477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 338.662409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 338.680236] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 343.109454] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 352.772554] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 358.403364] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 364.662834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 364.961570] Lustre: lustre-OST0000: new disk, initializing [ 364.963944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 364.968678] Lustre: 17024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 365.043221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 366.515077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 366.530173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 366.652086] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 371.048695] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 381.052537] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 386.377223] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 386.572593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 391.936320] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 392.196308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 397.288298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.289652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 397.298648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 397.301914] Lustre: Skipped 1 previous similar message [ 397.328765] LustreError: Skipped 1 previous similar message [ 400.802591] Lustre: server umount lustre-OST0000 complete [ 405.488535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 405.500920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.523838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 405.531248] Lustre: Skipped 1 previous similar message [ 410.982420] Lustre: server umount lustre-MDT0000 complete [ 412.647659] LustreError: 15063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.679390] LustreError: 15063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 414.733272] LustreError: 15861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782756279 with bad export cookie 16079032352655500219 [ 414.740586] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 414.740725] LustreError: 15861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 415.157409] Lustre: server umount lustre-MDT0001 complete [ 418.632422] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 14:04:41 (1782756281) [ 425.314939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 425.889889] LustreError: 19290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 425.946393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.835176] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 436.199462] LustreError: 19291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 436.225357] LustreError: 19291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 439.623112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.312604] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 452.195150] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 458.105604] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 465.541313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 472.582470] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 480.534479] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 491.490603] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 491.493521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.499435] LustreError: Skipped 1 previous similar message [ 491.508607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 491.509869] Lustre: Skipped 2 previous similar messages [ 491.523921] Lustre: Skipped 3 previous similar messages [ 493.924334] Lustre: server umount lustre-OST0000 complete [ 501.729051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 501.729704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.741597] Lustre: Skipped 1 previous similar message [ 503.552265] Lustre: server umount lustre-MDT0000 complete [ 506.865208] LustreError: 19285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.908470] LustreError: 19285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 507.993908] LustreError: 19273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782756373 with bad export cookie 16079032352655501101 [ 508.004879] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 509.010852] Lustre: server umount lustre-MDT0001 complete [ 528.013372] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 532.426118] Key type lgssc unregistered [ 532.963773] LNet: 22827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.975702] LNetError: 22827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 533.009359] LNet: Removed LNI 192.168.202.128@tcp [ 534.932814] Key type .llcrypt unregistered [ 534.937576] Key type ._llcrypt unregistered [ 566.762879] Key type ._llcrypt registered [ 566.764283] Key type .llcrypt registered [ 566.884614] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 568.739089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 568.944234] alg: No test for adler32 (adler32-zlib) [ 570.177259] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 570.555184] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 572.247298] Key type lgssc registered [ 573.436304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 582.955733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 582.977185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.511692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.427990] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 593.893021] LustreError: 24241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.919399] LustreError: 24241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 599.011178] LustreError: 24242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.436129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.873112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 606.349336] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 615.211953] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 622.335452] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 630.011827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 630.526967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 638.249550] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 648.959869] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 654.076582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 669.669869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 669.679977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 669.687291] Lustre: Skipped 1 previous similar message [ 669.688390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 669.737812] Lustre: Skipped 1 previous similar message [ 674.811244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 674.821925] Lustre: Skipped 1 previous similar message [ 675.276033] Lustre: server umount lustre-OST0000 complete [ 679.918514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 679.938341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 679.961200] Lustre: Skipped 1 previous similar message [ 679.997469] Lustre: Skipped 1 previous similar message [ 685.033688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 685.038626] Lustre: Skipped 1 previous similar message [ 686.031606] Lustre: server umount lustre-MDT0000 complete [ 690.145484] LustreError: 24242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.191638] LustreError: 24242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 690.556280] LustreError: 24222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782756555 with bad export cookie 13216916582197379414 [ 690.562469] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 690.574533] LustreError: 24222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 691.187460] Lustre: server umount lustre-MDT0001 complete [ 705.590234] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 709.323438] Key type lgssc unregistered [ 709.644255] LNet: 28220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.655258] LNetError: 28220:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.684771] LNet: Removed LNI 192.168.202.128@tcp [ 710.750520] Key type .llcrypt unregistered [ 710.754296] Key type ._llcrypt unregistered [ 739.877992] Key type ._llcrypt registered [ 739.886166] Key type .llcrypt registered [ 740.062803] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 741.925075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 741.961616] alg: No test for adler32 (adler32-zlib) [ 743.181887] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 743.664268] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 745.375502] Key type lgssc registered [ 746.493853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 755.730942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 755.752987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 757.294036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.855358] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 767.473045] LustreError: 29636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 767.482557] LustreError: 29636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 768.484779] LustreError: 29637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.268821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.669524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 776.107743] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 783.517712] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 789.215795] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 804.574796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 804.915452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 812.868350] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 821.854330] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 829.489504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 839.659121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.669332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 839.672350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 839.687050] Lustre: Skipped 1 previous similar message [ 844.773580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 844.783312] Lustre: Skipped 2 previous similar messages [ 845.614500] Lustre: server umount lustre-OST0000 complete [ 853.472612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 853.484390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 853.515233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 855.011146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.749650] Lustre: server umount lustre-MDT0000 complete [ 860.138497] LustreError: 31333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.161912] LustreError: 31333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 861.211314] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782756726 with bad export cookie 4279602715560023333 [ 861.215286] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 861.239774] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 862.028618] Lustre: server umount lustre-MDT0001 complete [ 876.844219] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 880.775108] Key type lgssc unregistered [ 881.305070] LNet: 33538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.313108] LNetError: 33538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.342038] LNet: Removed LNI 192.168.202.128@tcp [ 882.797407] Key type .llcrypt unregistered [ 882.798588] Key type ._llcrypt unregistered [ 911.049095] Key type ._llcrypt registered [ 911.054590] Key type .llcrypt registered [ 911.176893] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 912.401662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.671205] alg: No test for adler32 (adler32-zlib) [ 913.674210] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 913.863053] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 915.591734] Key type lgssc registered [ 916.795804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 925.876711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 925.927830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.478099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.303985] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 936.930830] LustreError: 34956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 936.956045] LustreError: 34956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 942.059781] LustreError: 34957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 943.127269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.744786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 947.443086] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 956.374933] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 966.734939] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 981.137412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 982.328749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.141363] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 997.932192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1005.318488] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1018.339257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1018.340853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.361991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1018.364030] Lustre: Skipped 1 previous similar message [ 1018.369779] Lustre: Skipped 1 previous similar message [ 1020.649810] Lustre: server umount lustre-OST0000 complete [ 1028.579189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1028.588128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1028.607788] Lustre: Skipped 1 previous similar message [ 1028.634408] Lustre: Skipped 1 previous similar message [ 1033.696950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1033.705512] Lustre: Skipped 1 previous similar message [ 1038.824853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1038.843859] Lustre: Skipped 1 previous similar message [ 1040.351598] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1040.641306] Lustre: server umount lustre-MDT0000 complete [ 1043.937779] LustreError: 34957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.965440] LustreError: 34957:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1044.073609] LustreError: 34938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782756909 with bad export cookie 728868138950908455 [ 1044.084159] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.085398] LustreError: 34938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1044.518815] Lustre: server umount lustre-MDT0001 complete [ 1056.328421] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1059.551805] Key type lgssc unregistered [ 1059.848326] LNet: 38499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.852977] LNetError: 38499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.904685] LNet: Removed LNI 192.168.202.128@tcp [ 1061.813667] Key type .llcrypt unregistered [ 1061.819915] Key type ._llcrypt unregistered [ 1089.192237] Key type ._llcrypt registered [ 1089.194867] Key type .llcrypt registered [ 1089.310564] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1090.449373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1090.482898] alg: No test for adler32 (adler32-zlib) [ 1091.770685] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 1092.004254] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1093.863524] Key type lgssc registered [ 1095.142179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1104.665080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1104.733768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.603945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1112.358199] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1116.137067] LustreError: 39920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.168577] LustreError: 39920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1121.249638] LustreError: 39919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.674459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.076257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1128.276543] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1136.740519] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1142.420290] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1149.683868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1150.372180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.425083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1157.208324] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1165.541206] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1177.057778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1177.068286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.082793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1178.785542] Lustre: lustre-OST0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1179.114439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.117241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1182.441518] Lustre: server umount lustre-OST0000 complete [ 1203.526268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.531976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.542832] Lustre: Skipped 1 previous similar message [ 1203.561889] Lustre: Skipped 3 previous similar messages [ 1209.826047] LustreError: 39920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.929841] Lustre: server umount lustre-MDT0000 complete [ 1214.516388] LustreError: 39900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782757079 with bad export cookie 8760881986390444044 [ 1214.541094] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1221.071093] Lustre: server umount lustre-MDT0001 complete [ 1237.064106] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1241.017756] Key type lgssc unregistered [ 1241.588288] LNet: 43630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.600572] LNetError: 43630:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.635625] LNet: Removed LNI 192.168.202.128@tcp [ 1243.282935] Key type .llcrypt unregistered [ 1243.286591] Key type ._llcrypt unregistered [ 1271.031508] Key type ._llcrypt registered [ 1271.033604] Key type .llcrypt registered [ 1271.142864] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1272.687891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1272.777283] alg: No test for adler32 (adler32-zlib) [ 1274.010763] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 1274.356629] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1276.113061] Key type lgssc registered [ 1277.662667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1286.459229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1286.483730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.179205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.176563] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1298.406096] LustreError: 45045:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.427359] LustreError: 45045:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1303.442352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.523565] LustreError: 45046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.823251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1308.743975] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1318.350638] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1325.993112] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1338.884838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1339.635833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.659584] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1358.061687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1359.148951] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1368.558065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1368.562172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1368.578985] Lustre: Skipped 2 previous similar messages [ 1368.586524] Lustre: Skipped 1 previous similar message [ 1371.314842] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1371.333915] Lustre: Skipped 1 previous similar message [ 1373.017648] Lustre: server umount lustre-MDT0000 complete [ 1373.671558] LustreError: 46044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1373.698634] LustreError: 46044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1376.391932] LustreError: 45041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.669068] LustreError: 45026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782757242 with bad export cookie 12214783426347429676 [ 1377.687916] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1377.699082] LustreError: 45026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1378.786696] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.822762] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1381.527042] LustreError: 45040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.547021] LustreError: 45040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1381.557228] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [ 1384.676645] Lustre: server umount lustre-MDT0001 complete [ 1402.303565] Lustre: server umount lustre-OST0000 complete [ 1424.697029] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1426.955709] Key type lgssc unregistered [ 1427.162823] LNet: 48912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.167630] LNetError: 48912:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.185427] LNet: Removed LNI 192.168.202.128@tcp [ 1428.072154] Key type .llcrypt unregistered [ 1428.076083] Key type ._llcrypt unregistered [ 1459.723445] Key type ._llcrypt registered [ 1459.724949] Key type .llcrypt registered [ 1459.852239] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1461.677869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1461.787135] alg: No test for adler32 (adler32-zlib) [ 1462.810953] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 1463.132358] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1464.863197] Key type lgssc registered [ 1466.700770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1476.119298] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1476.146088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1491.808556] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1496.930247] LustreError: 50300:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 1523.616057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.760335] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1541.675164] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1707.427718] Lustre: server umount lustre-OST0000 complete [ 1728.633065] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1731.811260] Key type lgssc unregistered [ 1732.052806] LNet: 51977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.061090] LNetError: 51977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.077359] LNet: Removed LNI 192.168.202.128@tcp [ 1732.864649] Key type .llcrypt unregistered [ 1732.869904] Key type ._llcrypt unregistered [ 1764.027756] Key type ._llcrypt registered [ 1764.029245] Key type .llcrypt registered [ 1764.083398] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1765.032205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1765.051481] alg: No test for adler32 (adler32-zlib) [ 1765.960143] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 1766.181021] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1767.865139] Key type lgssc registered [ 1769.293849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1779.465111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1779.498144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.176336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.928216] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1791.459781] LustreError: 53391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.480381] LustreError: 53391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1796.589339] LustreError: 53392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1798.239317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.649184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1804.208780] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1812.185189] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1819.167699] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1826.304387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1826.763773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.869598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1834.409861] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1844.865144] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1848.290127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1848.296245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.304350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1848.326076] Lustre: Skipped 1 previous similar message [ 1848.357033] Lustre: Skipped 1 previous similar message [ 1849.758748] Lustre: server umount lustre-OST0000 complete [ 1854.944373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1854.955965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.983290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1858.530546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1858.531250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.656092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1863.667493] Lustre: Skipped 2 previous similar messages [ 1868.255857] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1868.616659] Lustre: server umount lustre-MDT0000 complete [ 1868.769592] LustreError: 54489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1868.788825] LustreError: 54489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1872.811163] LustreError: 53372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782757737 with bad export cookie 10011663136036879864 [ 1872.814100] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.823408] LustreError: 53372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1873.280992] Lustre: server umount lustre-MDT0001 complete [ 1885.112080] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1888.561442] Key type lgssc unregistered [ 1888.886376] LNet: 56924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.907432] LNetError: 56924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.957911] LNet: Removed LNI 192.168.202.128@tcp [ 1891.280440] Key type .llcrypt unregistered [ 1891.284101] Key type ._llcrypt unregistered [ 1918.297114] Key type ._llcrypt registered [ 1918.298602] Key type .llcrypt registered [ 1918.479251] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1919.720832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1919.919289] alg: No test for adler32 (adler32-zlib) [ 1921.033916] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 1921.267068] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1922.938164] Key type lgssc registered [ 1924.311412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1932.642170] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1932.658456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1948.385059] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1953.503324] LustreError: 58312:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 1980.255235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.857430] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1996.248519] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2004.450155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2005.029981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.087590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2009.754121] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2010.081521] LustreError: 58337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.096304] LustreError: 58337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2015.211603] LustreError: 58338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2015.240571] LustreError: 58338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2019.540983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.976069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2025.025933] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2034.142359] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2040.646624] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2045.410846] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2045.425181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.452085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2045.926647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2045.941182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2045.963807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2051.042654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2056.178321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2056.180632] Lustre: Skipped 1 previous similar message [ 2057.184651] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2057.356951] Lustre: server umount lustre-OST0000 complete [ 2079.269845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2079.276482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2079.290338] Lustre: Skipped 1 previous similar message [ 2079.293700] Lustre: Skipped 1 previous similar message [ 2086.369482] LustreError: 59863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.402021] LustreError: 59863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2087.634699] Lustre: server umount lustre-MDT0000 complete [ 2091.498034] LustreError: 59354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.531399] LustreError: 59354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2093.026057] LustreError: 58318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782757958 with bad export cookie 8799779124321228224 [ 2093.033573] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2093.042346] LustreError: 58318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2099.569075] Lustre: server umount lustre-MDT0001 complete [ 2112.657414] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2116.139520] Key type lgssc unregistered [ 2116.451791] LNet: 62062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.460883] LNetError: 62062:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.476826] LNet: Removed LNI 192.168.202.128@tcp [ 2117.444735] Key type .llcrypt unregistered [ 2117.447920] Key type ._llcrypt unregistered [ 2145.700308] Key type ._llcrypt registered [ 2145.705632] Key type .llcrypt registered [ 2145.824538] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2147.112383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2147.188822] alg: No test for adler32 (adler32-zlib) [ 2148.201744] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 2148.507082] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2150.192126] Key type lgssc registered [ 2152.071726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2161.977390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2162.010082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.650923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.100496] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2172.903067] LustreError: 63478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.925399] LustreError: 63478:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2178.018504] LustreError: 63479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2180.630851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2181.058578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2187.298390] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2199.005622] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2206.057057] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2215.419759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2215.864222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2221.595996] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2228.648589] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2233.274696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2246.636182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2246.654690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2246.683704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2248.672602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.679629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2252.279580] Lustre: server umount lustre-OST0000 complete [ 2257.894316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2257.909374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.947323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2257.965618] Lustre: Skipped 1 previous similar message [ 2262.879285] Lustre: server umount lustre-MDT0000 complete [ 2264.032893] LustreError: 65245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.042439] LustreError: 65245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2266.073836] LustreError: 63459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782758131 with bad export cookie 2417821745279660531 [ 2266.077983] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2266.081489] LustreError: 63459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2266.419858] Lustre: server umount lustre-MDT0001 complete [ 2277.662759] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2280.327986] Key type lgssc unregistered [ 2280.627479] LNet: 67037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.647799] LNetError: 67037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2280.668714] LNet: Removed LNI 192.168.202.128@tcp [ 2281.275139] Key type .llcrypt unregistered [ 2281.278081] Key type ._llcrypt unregistered [ 2310.897934] Key type ._llcrypt registered [ 2310.899459] Key type .llcrypt registered [ 2311.031319] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2313.067503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2313.215573] alg: No test for adler32 (adler32-zlib) [ 2314.527789] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 2314.880837] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2316.648145] Key type lgssc registered [ 2318.805876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2330.541792] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 14:36:33 (1782758193) [ 2338.837909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2338.863176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.628593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2345.397101] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2350.049759] LustreError: 68751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.122318] LustreError: 68751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2353.999197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.541802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2359.838951] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2370.320316] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2377.424515] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2384.693057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2385.227392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.249105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2392.943880] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2402.160375] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2408.921703] Lustre: Failing over lustre-MDT0000 [ 2409.198988] Lustre: server umount lustre-MDT0000 complete [ 2409.967191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.969846] LustreError: 68751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.979041] Lustre: Skipped 1 previous similar message [ 2410.017124] LustreError: 68751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2415.079713] LustreError: 69479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.100259] LustreError: 69479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2418.162366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2418.241346] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2418.341319] Lustre: *** cfs_fail_loc=135, val=0*** [ 2418.414294] LustreError: 71841:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2418.424349] LustreError: 71841:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.128@tcp: cfg command failed: rc = -2 [ 2418.444142] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2418.444142] [ 2418.454183] LustreError: MGC192.168.202.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2418.466904] LustreError: 71829:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2418.475735] LustreError: 71829:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2418.485720] LustreError: 71829:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2418.495714] LustreError: 71829:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2418.523352] Lustre: server umount lustre-MDT0000 complete [ 2418.530108] LustreError: 71829:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2420.199344] LustreError: 68752:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.217762] LustreError: 68752:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2425.320630] LustreError: 68751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.336207] LustreError: 68751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2425.732190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2425.957752] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.968262] LustreError: 68743:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2426.327948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2426.406517] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2430.600506] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2431.485534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2431.554787] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2431.617220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2432.513221] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2444.656443] Lustre: server umount lustre-OST0000 complete [ 2451.937028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2451.937036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.937041] Lustre: Skipped 1 previous similar message [ 2451.937618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2452.003933] Lustre: Skipped 1 previous similar message [ 2457.068626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2457.077218] Lustre: Skipped 1 previous similar message [ 2459.180569] Lustre: server umount lustre-MDT0000 complete [ 2461.664244] Lustre: 67818:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782758310/real 1782758310] req@00000000d34daca9 x1869357441399424/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782758326 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2461.697797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.710321] Lustre: Skipped 1 previous similar message [ 2461.724272] LustreError: 68747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.736635] LustreError: 68747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2463.269045] LustreError: 68733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782758328 with bad export cookie 12066987355482329211 [ 2463.270146] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2463.279297] LustreError: 68733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2463.558864] Lustre: server umount lustre-MDT0001 complete [ 2474.961639] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2478.456457] Key type lgssc unregistered [ 2478.887837] LNet: 73623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2478.894507] LNetError: 73623:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2478.911716] LNet: Removed LNI 192.168.202.128@tcp [ 2479.970231] Key type .llcrypt unregistered [ 2479.973122] Key type ._llcrypt unregistered [ 2507.261408] Key type ._llcrypt registered [ 2507.263068] Key type .llcrypt registered [ 2507.387416] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2508.088921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2508.096832] alg: No test for adler32 (adler32-zlib) [ 2509.128910] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 2509.310987] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2510.927326] Key type lgssc registered [ 2512.144657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2520.392478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2520.406427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.799837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.940260] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2531.300719] LustreError: 75035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.325758] LustreError: 75035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2536.418866] LustreError: 75036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.089563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.816736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2547.337229] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2557.015972] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2563.934600] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2571.725772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2572.413277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2581.096391] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2590.275806] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2590.861415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2597.788081] Lustre: Failing over lustre-MDT0000 [ 2598.393043] Lustre: server umount lustre-MDT0000 complete [ 2598.566430] LustreError: 75032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.451187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.462646] LustreError: 75030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.468526] Lustre: Skipped 2 previous similar messages [ 2601.493178] LustreError: 75030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2606.566657] LustreError: 75031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.600560] LustreError: 75031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2608.156987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2608.305621] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.416217] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2608.515145] LustreError: 78124:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2608.520564] LustreError: 78124:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.128@tcp: cfg command failed: rc = -9 [ 2608.529237] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2608.529237] [ 2608.535209] LustreError: MGC192.168.202.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2608.545718] LustreError: 78112:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2608.551717] LustreError: 78112:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2608.568064] LustreError: 78112:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2608.580214] LustreError: 78112:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2608.626505] Lustre: server umount lustre-MDT0000 complete [ 2608.629738] LustreError: 78112:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2615.002268] LustreError: 76569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.022896] LustreError: 76569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2616.362959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.642077] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2616.804202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2616.816294] Lustre: Skipped 2 previous similar messages [ 2617.029626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.109190] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2620.062686] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2621.443826] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2622.473862] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2622.543857] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2622.611097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2632.163710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2632.182353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2632.203820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2636.481897] Lustre: server umount lustre-OST0000 complete [ 2642.914458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2642.929096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2642.935516] Lustre: Skipped 1 previous similar message [ 2642.949287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.957242] Lustre: Skipped 2 previous similar messages [ 2647.519363] Lustre: server umount lustre-MDT0000 complete [ 2648.037031] LustreError: 75036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.064521] LustreError: 75036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2651.444634] LustreError: 75017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782758516 with bad export cookie 6640328045310602384 [ 2651.456561] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2651.806926] Lustre: server umount lustre-MDT0001 complete [ 2663.493361] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2666.072504] Key type lgssc unregistered [ 2666.377965] LNet: 79904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2666.382518] LNetError: 79904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2666.397810] LNet: Removed LNI 192.168.202.128@tcp [ 2667.335039] Key type .llcrypt unregistered [ 2667.336668] Key type ._llcrypt unregistered [ 2680.345732] Key type ._llcrypt registered [ 2680.347972] Key type .llcrypt registered [ 2680.437567] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2680.853406] Key type .llcrypt unregistered [ 2680.858112] Key type ._llcrypt unregistered [ 2703.713084] Key type ._llcrypt registered [ 2703.717249] Key type .llcrypt registered [ 2703.890678] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2705.661265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2705.711091] alg: No test for adler32 (adler32-zlib) [ 2706.904060] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 2707.238958] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2709.607137] Key type lgssc registered [ 2711.229734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2722.245314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2722.285959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2724.019031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2730.461820] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2734.564444] LustreError: 81806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.584040] LustreError: 81806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2739.682418] LustreError: 81807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.435805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.883102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2748.297252] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2757.640047] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2764.990539] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2772.720645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.074849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.844418] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2788.648596] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2795.665232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2809.312833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2809.318834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2809.328701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2811.419428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2811.419945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2811.425443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.470104] Lustre: Skipped 1 previous similar message [ 2814.138789] Lustre: server umount lustre-OST0000 complete [ 2820.577840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2820.585462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2820.599819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.654842] Lustre: server umount lustre-MDT0000 complete [ 2826.720918] LustreError: 81807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.733499] LustreError: 81807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2829.813599] LustreError: 81786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782758694 with bad export cookie 17162743989300099422 [ 2829.824180] LustreError: 81786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2829.825520] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2830.213712] Lustre: server umount lustre-MDT0001 complete [ 2840.621809] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2843.259784] Key type lgssc unregistered [ 2843.570993] LNet: 85365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2843.587117] LNetError: 85365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2843.613076] LNet: Removed LNI 192.168.202.128@tcp [ 2844.306830] Key type .llcrypt unregistered [ 2844.309381] Key type ._llcrypt unregistered [ 2870.475524] Key type ._llcrypt registered [ 2870.481360] Key type .llcrypt registered [ 2870.654344] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2872.633825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2872.663214] alg: No test for adler32 (adler32-zlib) [ 2873.728899] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 2873.945512] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2875.632046] Key type lgssc registered [ 2876.926276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2885.035107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2885.064980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.640556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.143330] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2895.854463] LustreError: 86779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2895.876556] LustreError: 86779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2900.961125] LustreError: 86780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2901.480780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.757657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2906.852867] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2915.204495] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2922.207822] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2929.691581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2929.922682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.992478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2937.608766] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2945.889795] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2954.141992] Lustre: server umount lustre-OST0000 complete [ 2958.309255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2958.314415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2958.324155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2961.892826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2961.900790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2967.014439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2967.023562] Lustre: Skipped 2 previous similar messages [ 2972.128293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2972.128536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.128909] Lustre: 86092:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782758821/real 1782758821] req@000000004fc6abd1 x1869358027533952/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782758837 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2972.129218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.232162] Lustre: Skipped 1 previous similar message [ 2972.649596] Lustre: server umount lustre-MDT0000 complete [ 2977.249226] LustreError: 86780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.302093] LustreError: 86780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2977.837196] LustreError: 86761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782758842 with bad export cookie 16126723226862198851 [ 2977.847739] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2977.852603] LustreError: 86761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2978.190331] Lustre: server umount lustre-MDT0001 complete [ 2990.482483] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2992.753165] Key type lgssc unregistered [ 2992.977508] LNet: 90314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2992.984982] LNetError: 90314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2993.005254] LNet: Removed LNI 192.168.202.128@tcp [ 2993.716993] Key type .llcrypt unregistered [ 2993.719696] Key type ._llcrypt unregistered [ 3018.621489] Key type ._llcrypt registered [ 3018.624978] Key type .llcrypt registered [ 3018.770605] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3020.184486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3020.325234] alg: No test for adler32 (adler32-zlib) [ 3021.393314] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 3021.620240] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3023.311173] Key type lgssc registered [ 3024.633821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3033.086935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3033.124323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.735990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.863600] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3044.334377] LustreError: 91728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3044.353677] LustreError: 91728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3048.915319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.408918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3054.139939] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3066.514754] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3073.141059] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3083.664423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3084.315439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.533567] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3104.848706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3105.739607] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3124.925464] Lustre: server umount lustre-OST0000 complete [ 3130.337811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.347110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.354986] Lustre: Skipped 1 previous similar message [ 3135.458150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3135.463893] Lustre: Skipped 2 previous similar messages [ 3139.762261] Lustre: server umount lustre-MDT0000 complete [ 3140.579994] LustreError: 91728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.591506] LustreError: 91728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3141.151534] Lustre: 91040:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782758990/real 1782758990] req@00000000a0909ff1 x1869358181677440/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782759006 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3141.199459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3144.057692] LustreError: 91709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782759009 with bad export cookie 7432268988842665880 [ 3144.074637] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3144.078039] LustreError: 91709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3144.618565] Lustre: server umount lustre-MDT0001 complete [ 3155.668220] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3158.358195] Key type lgssc unregistered [ 3158.715688] LNet: 95285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3158.723150] LNetError: 95285:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3158.735748] LNet: Removed LNI 192.168.202.128@tcp [ 3159.679662] Key type .llcrypt unregistered [ 3159.681318] Key type ._llcrypt unregistered [ 3180.718150] Key type ._llcrypt registered [ 3180.723526] Key type .llcrypt registered [ 3180.836211] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3182.007950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3182.038876] alg: No test for adler32 (adler32-zlib) [ 3183.187310] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 3183.437118] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3185.143198] Key type lgssc registered [ 3186.414055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3193.939309] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3193.962032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3209.638501] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3214.751920] LustreError: 96651:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 3241.439514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.362909] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3254.450479] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3259.683418] Lustre: server umount lustre-OST0000 complete [ 3266.314657] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 14:52:10 (1782759130) [ 3271.745841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.079945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3275.718742] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3277.288688] LustreError: 98289:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3277.309322] LustreError: 98289:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3282.404529] LustreError: 98290:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.474297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3283.855328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3287.909431] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3294.518784] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3299.143975] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3304.796371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3305.035241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3309.925296] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3316.007266] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3317.367302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3333.095370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.099456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3333.110751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3333.133223] Lustre: Skipped 1 previous similar message [ 3336.540107] Lustre: server umount lustre-OST0000 complete [ 3340.255854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3340.258675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.281030] Lustre: Skipped 1 previous similar message [ 3340.285494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.332838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.335678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3346.231750] Lustre: server umount lustre-MDT0000 complete [ 3348.455289] LustreError: 99721:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3348.470909] LustreError: 99721:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3349.456748] LustreError: 98271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782759214 with bad export cookie 3686908896154328805 [ 3349.471602] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3349.475468] LustreError: 98271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3349.757585] Lustre: server umount lustre-MDT0001 complete [ 3357.235294] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3359.293439] Key type lgssc unregistered [ 3359.512572] LNet: 101978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3359.517712] LNetError: 101978:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3359.529734] LNet: Removed LNI 192.168.202.128@tcp [ 3360.258537] Key type .llcrypt unregistered [ 3360.260457] Key type ._llcrypt unregistered [ 3389.590551] Key type ._llcrypt registered [ 3389.592504] Key type .llcrypt registered [ 3389.675361] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3390.329058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3390.347240] alg: No test for adler32 (adler32-zlib) [ 3391.406316] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 3391.651883] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3393.303187] Key type lgssc registered [ 3394.054470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3399.382985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3399.403575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3400.740399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.591992] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3408.854388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.078508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3411.499043] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3416.285906] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3419.745856] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3423.949319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3424.173054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.191861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3427.323310] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3431.021177] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3444.705958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3444.711261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3444.730609] LustreError: Skipped 1 previous similar message [ 3444.731796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3444.737622] Lustre: Skipped 1 previous similar message [ 3447.306840] Lustre: server umount lustre-OST0000 complete [ 3449.824695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.825480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.839619] Lustre: Skipped 2 previous similar messages [ 3449.847990] Lustre: Skipped 1 previous similar message [ 3454.944087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.947814] Lustre: Skipped 1 previous similar message [ 3455.505192] Lustre: server umount lustre-MDT0000 complete [ 3457.727700] LustreError: 104124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782759322 with bad export cookie 2209513642907275817 [ 3457.734364] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3457.737625] LustreError: 104124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3457.944581] Lustre: server umount lustre-MDT0001 complete [ 3464.474605] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3466.121614] Key type lgssc unregistered [ 3466.292772] LNet: 107672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3466.296675] LNetError: 107672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3466.305201] LNet: Removed LNI 192.168.202.128@tcp [ 3466.801124] Key type .llcrypt unregistered [ 3466.802555] Key type ._llcrypt unregistered [ 3477.979172] Key type ._llcrypt registered [ 3477.980945] Key type .llcrypt registered [ 3478.059825] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3478.591893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3478.603729] alg: No test for adler32 (adler32-zlib) [ 3479.503507] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 3479.642659] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3481.231568] Key type lgssc registered [ 3481.923925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3486.463834] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3486.478526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3502.047399] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3507.167317] LustreError: 108895:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 3533.791246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3537.055839] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3541.129627] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3544.540459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.693702] LustreError: MGC192.168.202.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3544.699396] LustreError: 109919:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3544.702067] LustreError: 109919:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3544.706756] LustreError: 109919:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3544.743683] Lustre: server umount lustre-MDT0000 complete [ 3544.745151] LustreError: 109919:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3557.915738] Lustre: server umount lustre-OST0000 complete [ 3563.413523] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 3568.021255] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3586.886038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3587.007749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3587.030131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3587.073075] Lustre: lustre-MDT0000: new disk, initializing [ 3587.117997] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3587.126975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3589.043366] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3595.409053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.473373] Lustre: 112705:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3595.513675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3595.516703] Lustre: Skipped 1 previous similar message [ 3595.568881] Lustre: lustre-MDT0001: new disk, initializing [ 3595.618304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3595.648931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3595.660846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3597.682744] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3603.105680] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3606.060318] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3609.168926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3609.291149] Lustre: lustre-OST0000: new disk, initializing [ 3609.293403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3609.296277] Lustre: 114670:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3609.344614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.999844] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3616.813152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3616.822039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3616.850383] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3616.882102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3618.783347] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3618.892408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3620.968275] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3621.069054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3622.371656] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3622.374988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3622.380411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3626.978831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3626.979449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3626.981046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3626.992155] Lustre: Skipped 1 previous similar message [ 3632.098139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3632.100623] Lustre: Skipped 1 previous similar message [ 3636.191080] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3636.254398] Lustre: server umount lustre-OST0000 complete [ 3641.823875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3641.826813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.832485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3647.461226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3647.463644] Lustre: Skipped 3 previous similar messages [ 3652.063883] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3652.156982] Lustre: server umount lustre-MDT0000 complete [ 3655.982185] LustreError: 112697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782759521 with bad export cookie 14362517209784820543 [ 3655.984058] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3655.992715] LustreError: 112697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3656.164190] Lustre: server umount lustre-MDT0001 complete [ 3664.399656] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 14:58:47 (1782759527) [ 3669.456428] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3697.276744] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 3711.989279] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3771.909667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3772.297285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3772.388807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3772.631739] Lustre: lustre-MDT0000: new disk, initializing [ 3772.796404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3772.832809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3780.343813] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3794.204559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3794.408172] Lustre: 119598:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3794.509799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3794.515202] Lustre: Skipped 1 previous similar message [ 3794.637460] Lustre: lustre-MDT0001: new disk, initializing [ 3794.816544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3794.897148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3794.900701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3800.891974] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3811.747966] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3818.288761] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3825.868386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3826.213249] Lustre: lustre-OST0000: new disk, initializing [ 3826.220761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3826.227423] Lustre: 121578:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3826.321310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3833.303302] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3836.026138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3836.039773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3836.125462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3844.374976] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3849.132288] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3849.388441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3855.206960] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3855.369805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3861.984779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3861.988771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3861.991394] LustreError: Skipped 1 previous similar message [ 3861.996637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3862.002157] Lustre: Skipped 2 previous similar messages [ 3862.026658] Lustre: Skipped 1 previous similar message [ 3863.915438] Lustre: server umount lustre-OST0000 complete [ 3871.714817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3871.725832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3874.568389] Lustre: server umount lustre-MDT0000 complete [ 3877.367363] LustreError: 119611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3877.406468] LustreError: 119611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3880.017810] LustreError: 121576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782759745 with bad export cookie 14362517209784821404 [ 3880.021370] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.043798] LustreError: 121576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3880.793915] Lustre: server umount lustre-MDT0001 complete [ 3892.525061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.080924] LustreError: 123796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3893.161042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3898.351889] LustreError: 123797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3898.362179] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3903.465230] LustreError: 123796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.577960] LustreError: 123797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3909.582614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.332463] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3924.358394] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3931.651070] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3938.671321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3938.726627] LustreError: 125618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.273245] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3954.372600] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3961.806934] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3969.505434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3969.517234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.530856] Lustre: Skipped 1 previous similar message [ 3969.534801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3969.547477] Lustre: Skipped 3 previous similar messages [ 3972.893224] Lustre: server umount lustre-OST0000 complete [ 3983.776354] Lustre: server umount lustre-MDT0000 complete [ 3985.888327] LustreError: 124772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3985.902288] LustreError: 124772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3988.164689] LustreError: 123779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782759853 with bad export cookie 14362517209784822293 [ 3988.172372] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3988.187565] LustreError: 123779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3988.966768] Lustre: server umount lustre-MDT0001 complete [ 4002.003832] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 4005.247388] Key type lgssc unregistered [ 4005.522708] LNet: 127427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4005.529803] LNetError: 127427:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4005.556249] LNet: Removed LNI 192.168.202.128@tcp [ 4006.640450] Key type .llcrypt unregistered [ 4006.642164] Key type ._llcrypt unregistered [ 4028.338581] Key type ._llcrypt registered [ 4028.340756] Key type .llcrypt registered [ 4028.479972] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4046.797698] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4048.556989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4048.641724] alg: No test for adler32 (adler32-zlib) [ 4049.922322] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 4050.111854] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 4051.767808] Key type lgssc registered [ 4053.234111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4096.602081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4096.656376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4097.985715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4098.033221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4098.130361] Lustre: lustre-MDT0000: new disk, initializing [ 4098.229037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.277166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4103.335198] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4115.410537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.543808] Lustre: 130587:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4115.603741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4115.607680] Lustre: Skipped 1 previous similar message [ 4115.695174] Lustre: lustre-MDT0001: new disk, initializing [ 4115.834923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4115.894772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4115.920120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4121.350214] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4131.642301] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4139.079543] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4146.670770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4147.108222] Lustre: lustre-OST0000: new disk, initializing [ 4147.117421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4147.133785] Lustre: 132557:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4147.316299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4154.068633] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4157.017730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4157.028344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4157.151224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4163.769653] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4168.677739] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4168.833928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4173.283426] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4173.584391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4177.888832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4177.894536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.900794] LustreError: Skipped 1 previous similar message [ 4177.935500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4181.684650] Lustre: server umount lustre-OST0000 complete [ 4187.617610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4187.629685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.646753] Lustre: Skipped 1 previous similar message [ 4187.661168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.674487] Lustre: Skipped 1 previous similar message [ 4191.846370] Lustre: server umount lustre-MDT0000 complete [ 4195.874775] LustreError: 130579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782760061 with bad export cookie 4320870463534114423 [ 4195.878929] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4195.892725] LustreError: 130579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4196.252213] Lustre: server umount lustre-MDT0001 complete [ 4204.670261] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 15:07:48 (1782760068) [ 4212.039284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4212.462431] LustreError: 134967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4212.484704] LustreError: 134967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4212.541165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4217.809344] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4217.828143] LustreError: 134968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.934728] LustreError: 134967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4226.848710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.068333] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4227.079530] Lustre: Skipped 2 previous similar messages [ 4227.256785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4232.652929] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4240.593170] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4246.373680] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4252.391206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.395252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4252.399339] Lustre: Skipped 2 previous similar messages [ 4252.410054] Lustre: Skipped 1 previous similar message [ 4254.712208] Lustre: server umount lustre-MDT0000 complete [ 4257.761729] LustreError: 134967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4257.782833] LustreError: 134967:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4258.326924] LustreError: 134950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782760123 with bad export cookie 4320870463534115305 [ 4258.333506] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4258.344613] LustreError: 134950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4258.904633] Lustre: server umount lustre-MDT0001 complete [ 4268.188693] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 15:08:51 (1782760131) [ 4275.160307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4290.658797] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4295.775554] LustreError: 137382:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 4321.568771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4328.689254] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4337.283590] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4339.572294] Lustre: server umount lustre-OST0000 complete [ 4347.799284] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 15:10:11 (1782760211) [ 4355.918719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4356.421054] LustreError: 138830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.504985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4361.076530] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4366.819224] LustreError: 138831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4366.846281] LustreError: 138831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4370.157828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4375.505258] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4383.267388] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4389.147804] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4397.004564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4397.415747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.420489] Lustre: Skipped 1 previous similar message [ 4403.668189] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4411.395348] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4426.213707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.215392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4426.229931] Lustre: Skipped 2 previous similar messages [ 4426.245754] Lustre: Skipped 2 previous similar messages [ 4430.216435] Lustre: server umount lustre-MDT0000 complete [ 4431.329419] LustreError: 138831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.367589] LustreError: 138831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4434.406936] LustreError: 138812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782760299 with bad export cookie 4320870463534115970 [ 4434.427717] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4434.728674] Lustre: server umount lustre-MDT0001 complete [ 4448.444918] Lustre: server umount lustre-OST0000 complete [ 4457.431959] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 15:12:00 (1782760320) [ 4464.402982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4464.923092] LustreError: 142496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4465.022375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4469.784165] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4478.975384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.082942] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4493.981927] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4500.870720] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4508.373671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4510.783365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4515.194799] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4522.541224] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4527.689302] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4527.995292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4533.202780] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4533.482526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4536.290990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4536.307650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4536.334737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4541.733434] Lustre: server umount lustre-OST0000 complete [ 4549.615651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.640915] Lustre: Skipped 2 previous similar messages [ 4552.598689] Lustre: server umount lustre-MDT0000 complete [ 4554.721503] LustreError: 144089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.743646] LustreError: 144089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4557.746907] LustreError: 142478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782760422 with bad export cookie 4320870463534117034 [ 4557.750660] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4557.767076] LustreError: 142478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4559.059230] Lustre: server umount lustre-MDT0001 complete [ 4571.195102] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 15:13:54 (1782760434) [ 4578.407065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4594.079735] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4599.199447] LustreError: 146654:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 4624.931389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4624.934698] Lustre: Skipped 2 previous similar messages [ 4632.622669] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4642.283881] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4650.603340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.063308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4656.696415] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4668.766170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4673.415720] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4682.006886] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4690.196101] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4696.110265] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4696.335939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4702.657910] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4702.918345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4710.371349] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4710.375095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4710.379292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4710.387034] Lustre: Skipped 2 previous similar messages [ 4710.406099] Lustre: Skipped 6 previous similar messages [ 4711.926557] Lustre: server umount lustre-MDT0000 complete [ 4715.510313] LustreError: 146679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4715.559929] LustreError: 146679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 4716.477161] LustreError: 146660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782760581 with bad export cookie 4320870463534118063 [ 4716.484292] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4716.502955] LustreError: 146660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4717.071884] Lustre: server umount lustre-MDT0001 complete [ 4731.480254] Lustre: server umount lustre-OST0000 complete [ 4742.975953] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 15:16:45 (1782760605) [ 4752.277040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4768.223813] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4773.343704] LustreError: 150855:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 4805.569571] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4814.298306] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4822.348635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4822.935324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4827.892486] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4837.235286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.876227] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4850.607217] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4857.773681] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4865.676522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4865.949537] Lustre: lustre-OST0001: new disk, initializing [ 4865.962910] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4865.985682] Lustre: 153729:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4872.796137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4872.804058] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4872.966496] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4873.966807] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4883.702340] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4889.347587] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4889.659329] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4895.620559] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4896.121939] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4898.793044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4898.816038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4898.822377] Lustre: Skipped 1 previous similar message [ 4898.839135] Lustre: Skipped 1 previous similar message [ 4904.891209] Lustre: server umount lustre-OST0000 complete [ 4914.163065] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4924.383437] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4924.666529] Lustre: server umount lustre-OST0001 complete [ 4942.303665] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4942.590820] Lustre: server umount lustre-MDT0000 complete [ 4947.167404] LustreError: 153728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782760812 with bad export cookie 4320870463534119015 [ 4947.172673] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.180527] LustreError: 153728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4947.648887] Lustre: server umount lustre-MDT0001 complete [ 4975.319160] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 15:20:38 (1782760838) [ 4977.623765] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4979.898965] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4983.986885] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 15:20:45 (1782760845) [ 4995.498862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.973424] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4996.011750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4996.262499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.272253] Lustre: Skipped 6 previous similar messages [ 5001.973542] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5011.543119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.763483] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5011.808372] Lustre: 157293:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5017.633990] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5025.642706] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5031.795738] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5039.785438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5039.952075] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5046.647709] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5047.436073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5053.718379] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5058.206866] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5058.398634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5063.111586] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5063.330161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5067.744708] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5067.751379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5067.767975] Lustre: Skipped 4 previous similar messages [ 5067.776323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5067.782341] Lustre: Skipped 14 previous similar messages [ 5071.359384] Lustre: server umount lustre-OST0000 complete [ 5091.802449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5093.287119] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5093.297599] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5098.714595] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5106.086805] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5108.654507] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5108.664486] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5110.395773] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5110.547373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5114.328605] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5114.650700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5120.066508] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9897c95b4000.ost_server_uuid 50 [ 5121.771317] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9897c95b4000.ost_server_uuid in FULL state after 0 sec [ 5122.025975] LustreError: 157298:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.28@tcp) returned error from blocking AST (req@000000000538146c x1869359260899840 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000003981c3f3/0x3bf6d0cb2b6dc1c1 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.28@tcp remote: 0x8f5b716953f82d85 expref: 5 pid: 158047 timeout: 5222 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5122.064344] LustreError: lustre-MDT0000: A client on nid 192.168.202.28@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5122.083360] LustreError: 157288:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.202.28@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000003981c3f3/0x3bf6d0cb2b6dc1c1 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.28@tcp remote: 0x8f5b716953f82d85 expref: 6 pid: 158047 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5128.166374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5128.171164] LustreError: Skipped 1 previous similar message [ 5133.122678] Lustre: server umount lustre-OST0000 complete [ 5143.429239] Lustre: server umount lustre-MDT0000 complete [ 5144.545261] LustreError: 157303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5144.556905] LustreError: 157303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 5147.555810] LustreError: 157283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782761012 with bad export cookie 4320870463534120198 [ 5147.557138] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5147.566059] LustreError: 157283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5153.839111] Lustre: server umount lustre-MDT0001 complete [ 5168.175256] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 5171.067711] Key type lgssc unregistered [ 5171.366322] LNet: 163406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5171.383757] LNetError: 163406:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5171.408983] LNet: Removed LNI 192.168.202.128@tcp [ 5172.563829] Key type .llcrypt unregistered [ 5172.566932] Key type ._llcrypt unregistered [ 5197.234679] Key type ._llcrypt registered [ 5197.236250] Key type .llcrypt registered [ 5197.384477] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5198.712845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5198.777943] alg: No test for adler32 (adler32-zlib) [ 5199.924551] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 5200.134477] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 5201.792095] Key type lgssc registered [ 5203.101679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5209.929436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5209.943185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5211.321255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5216.278181] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5221.857600] LustreError: 164819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5221.873448] LustreError: 164819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5224.383366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.721466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5228.692036] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5236.474930] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5241.960614] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5248.253385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5248.635306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5254.135724] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5257.834122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5260.520486] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5266.047208] Lustre: Failing over lustre-MDT0000 [ 5266.402654] Lustre: server umount lustre-MDT0000 complete [ 5267.643314] LustreError: 164815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5268.449612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5268.457859] Lustre: Skipped 2 previous similar messages [ 5273.568899] LustreError: 166775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5273.575066] LustreError: 166775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5278.554356] LustreError: 164798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.28@tcp arrived at 1782761143 with bad export cookie 4193301837759109891 [ 5278.575911] LustreError: 164798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5278.701915] LustreError: 164820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5278.736407] LustreError: 164820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5283.809923] LustreError: 164819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5283.839694] LustreError: 164819:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5284.841468] Lustre: 164133:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782761133/real 1782761133] req@0000000031b1d7ca x1869360466519424/t0(0) o400->MGC192.168.202.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782761149 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5284.887338] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5288.233907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5295.078138] LustreError: 166775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5295.103339] LustreError: 166775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5295.485346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5295.556825] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5300.384903] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5300.710729] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5300.722198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5302.761979] Lustre: lustre-MDT0000: Denying connection for new client 2681f290-d4b7-4067-8f30-875a6fea021b (at 192.168.202.28@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5311.461680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5311.471857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.480737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5316.069242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5316.085465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5316.102897] Lustre: Skipped 1 previous similar message [ 5317.401973] Lustre: server umount lustre-OST0000 complete [ 5320.781759] LustreError: 168552:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5320.789765] Lustre: 167900:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5320.798611] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5320.817919] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5320.833735] LustreError: 167900:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e452720b x1869360466539520/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5320.855826] LustreError: 167900:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5320.864232] LustreError: 167900:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5320.874342] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5321.143910] Lustre: server umount lustre-MDT0000 complete [ 5321.185065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.195595] LustreError: 164814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5324.428623] LustreError: 164798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782761189 with bad export cookie 4193301837759109989 [ 5324.431958] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5324.438759] LustreError: 164798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5330.944660] Lustre: server umount lustre-MDT0001 complete [ 5342.318707] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 5344.822305] Key type lgssc unregistered [ 5345.117244] LNet: 169305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5345.128230] LNetError: 169305:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5346.150066] LNet: Removed LNI 192.168.202.128@tcp [ 5347.216402] Key type .llcrypt unregistered [ 5347.218248] Key type ._llcrypt unregistered [ 5371.390553] Key type ._llcrypt registered [ 5371.396565] Key type .llcrypt registered [ 5371.512933] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5372.811904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5372.877562] alg: No test for adler32 (adler32-zlib) [ 5374.022860] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 5374.331925] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 5376.023172] Key type lgssc registered [ 5377.337896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5385.102365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5385.134266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5386.689219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.000116] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5395.943342] LustreError: 170724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5395.976198] LustreError: 170724:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5399.841863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5400.136915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5404.953337] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5412.713890] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5419.490847] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5426.944245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5427.340324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5431.474196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5434.312988] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5441.930907] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5451.245442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5451.258927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.260438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.271750] Lustre: Skipped 1 previous similar message [ 5451.303238] Lustre: Skipped 1 previous similar message [ 5455.659981] Lustre: server umount lustre-OST0000 complete [ 5461.477729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5461.483192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5461.510374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5466.594192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5466.598972] Lustre: Skipped 2 previous similar messages [ 5471.725662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5471.728406] Lustre: Skipped 1 previous similar message [ 5474.272198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5474.427869] Lustre: server umount lustre-MDT0000 complete [ 5476.839617] LustreError: 170720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.865489] LustreError: 170720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5478.694291] LustreError: 170703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782761343 with bad export cookie 8886608187634483721 [ 5478.705899] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5479.138761] Lustre: server umount lustre-MDT0001 complete [ 5490.468814] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 5493.389540] Key type lgssc unregistered [ 5493.606076] LNet: 174256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5493.619569] LNetError: 174256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5493.634708] LNet: Removed LNI 192.168.202.128@tcp [ 5494.595079] Key type .llcrypt unregistered [ 5494.597493] Key type ._llcrypt unregistered [ 5531.564255] Key type ._llcrypt registered [ 5531.574715] Key type .llcrypt registered [ 5531.723595] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5532.859065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5533.014562] alg: No test for adler32 (adler32-zlib) [ 5534.200859] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 5534.491384] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 5536.167579] Key type lgssc registered [ 5537.410380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5546.689865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5546.714948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.298957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.385187] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5558.764343] LustreError: 175958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.788100] LustreError: 175958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5562.881306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.362324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5569.053943] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5577.295583] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5583.572976] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5591.935129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5592.377544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.408186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5599.483575] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5608.486379] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5619.172690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5619.178718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5619.201731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5622.419864] Lustre: server umount lustre-OST0000 complete [ 5629.410907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5629.412822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5629.420242] Lustre: Skipped 2 previous similar messages [ 5629.436599] Lustre: Skipped 2 previous similar messages [ 5632.976813] Lustre: server umount lustre-MDT0000 complete [ 5634.534274] LustreError: 175958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5634.553923] LustreError: 175958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5637.311917] LustreError: 175941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782761502 with bad export cookie 8358323307582139326 [ 5637.320022] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5637.324422] LustreError: 175941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5637.755263] Lustre: server umount lustre-MDT0001 complete [ 5653.122895] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 5656.427336] Key type lgssc unregistered [ 5656.689752] LNet: 179493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5656.695914] LNetError: 179493:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5657.768149] LNet: Removed LNI 192.168.202.128@tcp [ 5658.844214] Key type .llcrypt unregistered [ 5658.845835] Key type ._llcrypt unregistered [ 5685.005318] Key type ._llcrypt registered [ 5685.010090] Key type .llcrypt registered [ 5685.125973] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5686.795415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5686.825105] alg: No test for adler32 (adler32-zlib) [ 5688.302307] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 5688.697964] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 5690.399183] Key type lgssc registered [ 5692.006296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5705.100876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5705.138419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5706.421628] Lustre: *** cfs_fail_loc=135, val=0*** [ 5706.482441] LustreError: 181031:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5706.485547] LustreError: 181031:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.128@tcp: cfg command failed: rc = -2 [ 5706.490446] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5706.490446] [ 5706.498444] LustreError: MGC192.168.202.128@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5706.512527] LustreError: 180992:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5706.519671] LustreError: 180992:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5706.526174] LustreError: 180992:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5706.529857] LustreError: 180992:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5706.605750] Lustre: server umount lustre-MDT0000 complete [ 5706.607824] LustreError: 180992:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5714.068518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5714.655316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5720.136417] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5724.201216] LustreError: 181403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.210974] LustreError: 181403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5729.258370] LustreError: 181402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5730.406170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5730.830366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5735.979447] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5744.965882] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5751.486474] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5755.363122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.370687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5755.398928] Lustre: Skipped 1 previous similar message [ 5755.425037] Lustre: Skipped 1 previous similar message [ 5756.454736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5756.471120] Lustre: Skipped 1 previous similar message [ 5761.504899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5761.508126] Lustre: Skipped 1 previous similar message [ 5766.624767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5766.636397] Lustre: Skipped 1 previous similar message [ 5768.159876] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5768.400837] Lustre: server umount lustre-MDT0000 complete [ 5771.753653] LustreError: 181398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.781052] LustreError: 181398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5773.567892] LustreError: 181385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782761638 with bad export cookie 16939975832099410672 [ 5773.594511] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5774.183721] Lustre: server umount lustre-MDT0001 complete [ 5787.691927] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 5791.277798] Key type lgssc unregistered [ 5791.591271] LNet: 183725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5791.613544] LNetError: 183725:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5791.629946] LNet: Removed LNI 192.168.202.128@tcp [ 5792.877648] Key type .llcrypt unregistered [ 5792.879253] Key type ._llcrypt unregistered [ 5817.956568] Key type ._llcrypt registered [ 5817.959303] Key type .llcrypt registered [ 5818.055460] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 5818.496457] Key type .llcrypt unregistered [ 5818.498673] Key type ._llcrypt unregistered [ 5838.168425] Key type ._llcrypt registered [ 5838.177996] Key type .llcrypt registered [ 5838.322498] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5840.002296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5840.026870] alg: No test for adler32 (adler32-zlib) [ 5841.142977] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 5841.376814] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 5843.081144] Key type lgssc registered [ 5844.589151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5855.264882] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5855.296262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5870.943615] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5876.064254] LustreError: 185860:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 5902.689822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5912.889949] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5922.423683] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5930.640483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.105528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5931.190668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5936.112905] LustreError: 185886:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.146753] LustreError: 185886:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5937.193065] LustreError: 185885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5937.233947] LustreError: 185885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5937.389695] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5942.293202] LustreError: 186898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5942.341819] LustreError: 186898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5947.390273] LustreError: 186897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5947.438596] LustreError: 186897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5948.798870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5949.684278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5956.631683] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5966.392693] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5973.635775] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5977.059795] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5990.384814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5990.392933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5990.402092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5996.003426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5996.011142] Lustre: Skipped 2 previous similar messages [ 5999.015185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5999.032359] Lustre: Skipped 1 previous similar message [ 6004.191170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6004.193807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6004.213044] Lustre: Skipped 1 previous similar message [ 6004.345543] Lustre: server umount lustre-OST0000 complete [ 6009.320651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.327751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6009.332149] Lustre: Skipped 2 previous similar messages [ 6009.344311] Lustre: Skipped 1 previous similar message [ 6015.191823] Lustre: server umount lustre-MDT0000 complete [ 6019.490628] LustreError: 185867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782761884 with bad export cookie 814564882853833854 [ 6019.498978] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6019.510584] LustreError: 185867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6020.048958] Lustre: server umount lustre-MDT0001 complete [ 6033.128958] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6036.456973] Key type lgssc unregistered [ 6036.831726] LNet: 189878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6036.836758] LNetError: 189878:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6037.866257] LNet: Removed LNI 192.168.202.128@tcp [ 6038.956754] Key type .llcrypt unregistered [ 6038.959528] Key type ._llcrypt unregistered [ 6069.312574] Key type ._llcrypt registered [ 6069.321857] Key type .llcrypt registered [ 6069.566822] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6071.714599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6071.884927] alg: No test for adler32 (adler32-zlib) [ 6073.482620] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 6073.878442] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6075.647278] Key type lgssc registered [ 6077.439907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6086.110130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6086.143629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6087.477403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6092.861811] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6096.870783] LustreError: 191307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6096.888777] LustreError: 191307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6102.001862] LustreError: 191306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.892423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.374702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6109.268277] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6118.632432] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6127.042065] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6136.569860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6137.025388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6142.673518] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6151.819844] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6157.518453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6161.872339] Lustre: Failing over lustre-MDT0000 [ 6162.342499] Lustre: server umount lustre-MDT0000 complete [ 6162.933943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.954801] LustreError: 191301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.975220] LustreError: 191301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6168.041418] LustreError: 191306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.065567] LustreError: 191306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6173.175930] LustreError: 191301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6173.196955] LustreError: 191301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6179.297842] Lustre: 190619:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782762028/real 1782762028] req@00000000d60c3d57 x1869361382977536/t0(0) o400->MGC192.168.202.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782762044 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6179.329611] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6182.278630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.069571] LustreError: 193152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.095432] LustreError: 193152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6189.547626] LustreError: 190615:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a5502f7b x1869361382983680/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6189.789240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6189.831820] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6190.213883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6193.640588] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6195.199501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6195.220631] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6195.283539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6198.328962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6200.427902] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6215.653257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.659590] Lustre: Skipped 2 previous similar messages [ 6215.664638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.109194] Lustre: server umount lustre-OST0000 complete [ 6225.889626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6225.893079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6225.903490] Lustre: Skipped 1 previous similar message [ 6225.904516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6225.928198] Lustre: Skipped 2 previous similar messages [ 6227.396625] Lustre: server umount lustre-MDT0000 complete [ 6231.009747] LustreError: 191306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6231.029828] LustreError: 191306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6231.544647] LustreError: 191287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782762096 with bad export cookie 8696303066778252090 [ 6231.547976] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6231.555087] LustreError: 191287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6232.023715] Lustre: server umount lustre-MDT0001 complete [ 6245.011967] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6248.727648] Key type lgssc unregistered [ 6249.063223] LNet: 196069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6249.076787] LNetError: 196069:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6249.109355] LNet: Removed LNI 192.168.202.128@tcp [ 6250.163430] Key type .llcrypt unregistered [ 6250.172385] Key type ._llcrypt unregistered [ 6279.210331] Key type ._llcrypt registered [ 6279.212497] Key type .llcrypt registered [ 6279.342098] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6281.061318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6281.096421] alg: No test for adler32 (adler32-zlib) [ 6282.134866] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 6282.404499] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6284.055184] Key type lgssc registered [ 6285.115549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6294.707045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6294.728640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6296.338097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6301.701426] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6306.798306] LustreError: 197483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6306.828576] LustreError: 197483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6311.909581] LustreError: 197484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.181873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6312.754476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6318.445509] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6326.595465] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6334.193505] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6343.058943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6343.468553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6350.558554] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6359.240839] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6366.034103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6373.158981] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6377.783879] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6395.961465] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6409.190954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6409.204050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6409.219739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6412.256778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6412.266653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6412.269389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6413.518711] Lustre: server umount lustre-OST0000 complete [ 6420.449907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6420.458291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6420.473869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6420.481037] Lustre: Skipped 1 previous similar message [ 6422.496711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6422.498835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6427.621242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6427.628987] Lustre: Skipped 2 previous similar messages [ 6432.223351] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6432.465875] Lustre: server umount lustre-MDT0000 complete [ 6432.736765] LustreError: 198211:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6432.757583] LustreError: 198211:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6436.734129] LustreError: 197463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782762301 with bad export cookie 875858242890322996 [ 6436.748097] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6436.758616] LustreError: 197463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6437.039853] Lustre: server umount lustre-MDT0001 complete [ 6451.538575] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6454.820315] Key type lgssc unregistered [ 6455.132493] LNet: 201382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6455.143972] LNetError: 201382:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6455.171290] LNet: Removed LNI 192.168.202.128@tcp [ 6456.491948] Key type .llcrypt unregistered [ 6456.493855] Key type ._llcrypt unregistered [ 6478.753097] Key type ._llcrypt registered [ 6478.756846] Key type .llcrypt registered [ 6478.870666] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6479.906370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6479.941751] alg: No test for adler32 (adler32-zlib) [ 6481.615595] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 6482.083325] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6483.975221] Key type lgssc registered [ 6485.105991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6493.561967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6493.588055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6495.070511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.727370] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6505.445334] LustreError: 202649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6505.490854] LustreError: 202649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6509.684116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.328641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6515.491898] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6525.161954] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6531.678336] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6539.296397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6539.720540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6540.817473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6546.445160] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6555.199259] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6561.827795] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6573.736638] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6584.723045] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6592.913962] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6601.195176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.203897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6601.207446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6606.313020] LustreError: 205435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6606.337264] LustreError: 205435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6606.448379] Lustre: server umount lustre-OST0000 complete [ 6611.426507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.441057] Lustre: Skipped 1 previous similar message [ 6611.445158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6611.447664] Lustre: Skipped 2 previous similar messages [ 6616.545116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.550863] Lustre: Skipped 1 previous similar message [ 6621.664621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6621.670653] Lustre: Skipped 1 previous similar message [ 6625.248168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6625.490331] Lustre: server umount lustre-MDT0000 complete [ 6626.790991] LustreError: 202649:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6626.808718] LustreError: 202649:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6630.071221] LustreError: 202629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782762495 with bad export cookie 17620528600897736794 [ 6630.081685] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6630.084951] LustreError: 202629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6630.455721] Lustre: server umount lustre-MDT0001 complete [ 6643.821785] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6647.420165] Key type lgssc unregistered [ 6647.684173] LNet: 207574:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6647.693871] LNetError: 207574:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6647.719849] LNet: Removed LNI 192.168.202.128@tcp [ 6648.743365] Key type .llcrypt unregistered [ 6648.746865] Key type ._llcrypt unregistered [ 6674.387652] Key type ._llcrypt registered [ 6674.391726] Key type .llcrypt registered [ 6674.522612] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6675.786248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6675.803681] alg: No test for adler32 (adler32-zlib) [ 6676.896648] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 6677.151698] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6678.863219] Key type lgssc registered [ 6680.650790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6689.799102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6689.844789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.341736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6695.164645] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6701.543505] LustreError: 208988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6701.577602] LustreError: 208988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6703.165318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.558939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6707.827744] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6715.297115] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6720.732635] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6726.941222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6727.276279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6733.388694] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6736.491745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6741.555966] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6757.350289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6757.358414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6757.374484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6757.379671] Lustre: Skipped 1 previous similar message [ 6757.394499] Lustre: Skipped 1 previous similar message [ 6760.779798] Lustre: server umount lustre-OST0000 complete [ 6765.029103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6765.042239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6765.054443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6767.588698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6767.590034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.886971] Lustre: server umount lustre-MDT0000 complete [ 6772.713822] LustreError: 208988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6775.377332] LustreError: 208969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782762640 with bad export cookie 5974110811854431673 [ 6775.382430] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6775.386334] LustreError: 208969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6775.815950] Lustre: server umount lustre-MDT0001 complete [ 6787.224867] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6790.244759] Key type lgssc unregistered [ 6790.576519] LNet: 212672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6790.582736] LNetError: 212672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6790.613368] LNet: Removed LNI 192.168.202.128@tcp [ 6791.684186] Key type .llcrypt unregistered [ 6791.686115] Key type ._llcrypt unregistered [ 6807.592426] Key type ._llcrypt registered [ 6807.595449] Key type .llcrypt registered [ 6807.666356] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6808.710376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6808.774185] alg: No test for adler32 (adler32-zlib) [ 6809.832732] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 6810.034630] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6811.703177] Key type lgssc registered [ 6812.622457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6819.809250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6819.836270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6821.328403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6825.479541] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6831.604301] LustreError: 213918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6831.633984] LustreError: 213918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6836.711466] LustreError: 213919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6836.770080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.186823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6841.931212] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6851.321799] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6858.161368] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6865.113804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6865.627781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6866.681824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6872.424407] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6880.730900] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6897.645358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6897.648755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6897.669485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6897.676211] LustreError: Skipped 1 previous similar message [ 6897.726218] Lustre: Skipped 1 previous similar message [ 6900.931464] Lustre: server umount lustre-OST0000 complete [ 6907.874217] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6907.888780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6907.893280] Lustre: Skipped 2 previous similar messages [ 6912.099689] Lustre: server umount lustre-MDT0000 complete [ 6912.993804] LustreError: 214920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6913.024699] LustreError: 214920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6916.855815] LustreError: 214652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782762782 with bad export cookie 2522784043535874848 [ 6916.865256] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6916.866884] LustreError: 214652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6917.291259] Lustre: server umount lustre-MDT0001 complete [ 6929.368168] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6932.647823] Key type lgssc unregistered [ 6933.101721] LNet: 217601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6933.107852] LNetError: 217601:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6934.184077] LNet: Removed LNI 192.168.202.128@tcp [ 6935.314462] Key type .llcrypt unregistered [ 6935.316354] Key type ._llcrypt unregistered [ 6955.719878] Key type ._llcrypt registered [ 6955.723872] Key type .llcrypt registered [ 6955.912530] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6957.160704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6957.261290] alg: No test for adler32 (adler32-zlib) [ 6958.789377] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 6959.217953] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6960.991437] Key type lgssc registered [ 6962.771969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6974.063849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6974.090237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.048625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6981.414901] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6986.213196] LustreError: 218883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6986.252660] LustreError: 218883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6990.824844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6991.202589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6995.233780] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7003.198988] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7009.898540] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7016.696585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.229799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7021.374524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7023.879548] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7031.611821] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7041.773635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7041.832699] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7042.000497] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7048.668643] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7056.668274] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7058.685626] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7070.593422] Lustre: Permanently deactivating lustre-OST0001 [ 7070.610163] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7078.501963] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7089.772048] Lustre: Permanently reactivating lustre-OST0001 [ 7089.804899] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7089.813807] Lustre: Skipped 2 previous similar messages [ 7097.471497] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.490288] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7097.503256] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7097.524789] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7097.565608] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7100.278088] Lustre: server umount lustre-OST0001 complete [ 7102.972147] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7102.973453] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7102.986919] Lustre: Skipped 2 previous similar messages [ 7102.990630] LustreError: 221993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7103.033852] LustreError: 221993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7108.069935] LustreError: 221993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7108.071504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7108.098485] LustreError: 221993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7108.113173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7108.125092] Lustre: Skipped 1 previous similar message [ 7113.187515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7113.188537] LustreError: 221993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7113.194563] Lustre: Skipped 1 previous similar message [ 7113.219305] LustreError: 221993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7118.303372] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7118.309525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7118.313099] LustreError: 221993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7118.330933] Lustre: Skipped 1 previous similar message [ 7118.541793] Lustre: server umount lustre-OST0000 complete [ 7123.446171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7123.447196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7123.464967] Lustre: Skipped 2 previous similar messages [ 7128.547179] LustreError: 218883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7128.566858] LustreError: 218883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7128.637626] Lustre: server umount lustre-MDT0000 complete [ 7133.171883] LustreError: 218864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782762998 with bad export cookie 7125661340037821638 [ 7133.188303] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7133.190752] LustreError: 218864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7133.701515] Lustre: server umount lustre-MDT0001 complete [ 7147.172472] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7149.880247] Key type lgssc unregistered [ 7150.199830] LNet: 223956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7150.218145] LNetError: 223956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7150.239941] LNet: Removed LNI 192.168.202.128@tcp [ 7151.202072] Key type .llcrypt unregistered [ 7151.203910] Key type ._llcrypt unregistered [ 7191.302833] Key type ._llcrypt registered [ 7191.306934] Key type .llcrypt registered [ 7191.418889] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7192.698935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7192.773693] alg: No test for adler32 (adler32-zlib) [ 7193.893693] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 7194.092269] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7195.791191] Key type lgssc registered [ 7196.650859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7204.020218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7204.034713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.278675] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7205.296753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7205.436343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7210.118388] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7219.371566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7219.542295] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7219.594403] Lustre: 226159:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7219.954155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7225.435117] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7234.484489] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7240.878994] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7248.959693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7249.207887] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7249.491502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.563732] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7258.725300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7264.496689] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7268.446718] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7271.981446] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7277.666532] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7284.291194] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7299.544401] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7299.550284] Lustre: Skipped 1 previous similar message [ 7316.498694] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7316.512597] Lustre: Skipped 1 previous similar message [ 7353.002672] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7353.015198] Lustre: Skipped 4 previous similar messages [ 7417.362758] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7417.367283] Lustre: Skipped 7 previous similar messages [ 7438.306669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7438.318202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.346481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7442.912938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7442.918452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7442.925414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7444.496678] Lustre: server umount lustre-OST0000 complete [ 7450.595789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7450.599074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7450.631932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7450.634190] Lustre: Skipped 1 previous similar message [ 7453.155503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7453.174317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7453.184512] Lustre: Skipped 1 previous similar message [ 7455.496426] Lustre: server umount lustre-MDT0000 complete [ 7459.364897] LustreError: 226152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782763324 with bad export cookie 5766688976584300386 [ 7459.373156] LustreError: 226152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7459.374868] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7459.832977] Lustre: server umount lustre-MDT0001 complete [ 7472.302907] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7475.469111] Key type lgssc unregistered [ 7475.870621] LNet: 231010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7475.880631] LNetError: 231010:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7475.904765] LNet: Removed LNI 192.168.202.128@tcp [ 7476.807521] Key type .llcrypt unregistered [ 7476.811552] Key type ._llcrypt unregistered [ 7503.576114] Key type ._llcrypt registered [ 7503.580253] Key type .llcrypt registered [ 7503.820109] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7505.177654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7505.215648] alg: No test for adler32 (adler32-zlib) [ 7506.443805] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 7506.696443] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7508.383187] Key type lgssc registered [ 7509.538644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7517.830863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7517.851238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7519.337918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.003385] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7528.950715] LustreError: 232425:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7528.969701] LustreError: 232425:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7533.304778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7533.991861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7539.478717] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7548.361644] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7555.395841] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7562.746331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7563.257442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7564.283216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7569.430511] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7578.622842] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7600.106660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7600.113022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7600.123909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7600.133293] Lustre: Skipped 1 previous similar message [ 7600.163326] LustreError: Skipped 1 previous similar message [ 7601.536593] Lustre: server umount lustre-OST0000 complete [ 7610.337847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.342965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7610.356914] Lustre: Skipped 1 previous similar message [ 7610.378717] Lustre: Skipped 2 previous similar messages [ 7612.610627] Lustre: server umount lustre-MDT0000 complete [ 7615.458416] LustreError: 232420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7615.488138] LustreError: 232420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7617.436980] LustreError: 232406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782763482 with bad export cookie 7639286325775103781 [ 7617.440791] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7617.448418] LustreError: 232406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7617.891206] Lustre: server umount lustre-MDT0001 complete [ 7632.451704] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7635.480786] Key type lgssc unregistered [ 7635.739307] LNet: 236117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7635.743222] LNetError: 236117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7635.782112] LNet: Removed LNI 192.168.202.128@tcp [ 7636.900621] Key type .llcrypt unregistered [ 7636.905469] Key type ._llcrypt unregistered [ 7664.007884] Key type ._llcrypt registered [ 7664.010865] Key type .llcrypt registered [ 7664.180436] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7665.970537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7666.058057] alg: No test for adler32 (adler32-zlib) [ 7667.280445] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 7667.648916] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7669.375775] Key type lgssc registered [ 7670.627600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7943.251353] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7947.148937] Key type lgssc unregistered [ 7947.618974] LNet: 238139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7947.633484] LNetError: 238139:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7947.655539] LNet: Removed LNI 192.168.202.128@tcp [ 7948.882619] Key type .llcrypt unregistered [ 7948.885054] Key type ._llcrypt unregistered [ 7978.416952] Key type ._llcrypt registered [ 7978.420382] Key type .llcrypt registered [ 7978.562988] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 7994.570939] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7996.033894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7996.069721] alg: No test for adler32 (adler32-zlib) [ 7997.264684] Lustre: Lustre: Build Version: 2.17.54_83_g2cb683b [ 7997.515544] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7999.241083] Key type lgssc registered [ 8000.699143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8040.673198] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7824 sec ========== 16:11:43 (1782763903) [ 8043.052206] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:11:45 (1782763905) === [ 8047.876071] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:11:50 (1782763910) === [ 8095.637822] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 8099.433966] Key type lgssc unregistered [ 8099.774689] LNet: 243378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8099.788555] LNetError: 243378:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8099.801189] LNet: Removed LNI 192.168.202.128@tcp [ 8100.961178] Key type .llcrypt unregistered [ 8100.964136] Key type ._llcrypt unregistered