[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485261516 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003373] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008422] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011126] LSM: Security Framework initializing [ 0.012066] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.015008] *** VALIDATE selinux *** [ 0.023106] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028530] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030059] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032025] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033117] *** VALIDATE tmpfs *** [ 0.035195] *** VALIDATE proc *** [ 0.036231] *** VALIDATE cgroup *** [ 0.037009] *** VALIDATE cgroup2 *** [ 0.038268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041031] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.045015] debug: unmapping init [mem 0xffffffffb3259000-0xffffffffb3260fff] [ 0.048190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049722] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050026] ... version: 2 [ 0.051013] ... bit width: 48 [ 0.052012] ... generic registers: 4 [ 0.053014] ... value mask: 0000ffffffffffff [ 0.054015] ... max period: 00007fffffffffff [ 0.055016] ... fixed-purpose events: 3 [ 0.056013] ... event mask: 000000070000000f [ 0.058242] rcu: Hierarchical SRCU implementation. [ 0.060402] smp: Bringing up secondary CPUs ... [ 0.061604] x86: Booting SMP configuration: [ 0.062028] .... node #0, CPUs: #1 #2 #3 [ 0.065500] smp: Brought up 1 node, 4 CPUs [ 0.067014] smpboot: Max logical packages: 1 [ 0.068020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.263000] node 0 deferred pages initialised in 193ms [ 0.267463] devtmpfs: initialized [ 0.268246] x86/mm: Memory block size: 128MB [ 0.270798] gcov: version magic: 0x41383552 [ 0.271622] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.276108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.278395] pinctrl core: initialized pinctrl subsystem [ 0.280215] [ 0.280938] ************************************************************* [ 0.283021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.286016] ** ** [ 0.288012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.291017] ** ** [ 0.293011] ** This means that this kernel is built to expose internal ** [ 0.296021] ** IOMMU data structures, which may compromise security on ** [ 0.299021] ** your system. ** [ 0.301010] ** ** [ 0.304017] ** If you see this message and you are not debugging the ** [ 0.306018] ** kernel, report this immediately to your vendor! ** [ 0.309021] ** ** [ 0.311014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.313015] ************************************************************* [ 0.317869] NET: Registered protocol family 16 [ 0.320471] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.324083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.327084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.331146] cpuidle: using governor menu [ 0.332935] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.335599] PCI: Using configuration type 1 for base access [ 0.338139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.348132] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.351031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.356086] cryptd: max_cpu_qlen set to 1000 [ 0.358045] ACPI: Added _OSI(Module Device) [ 0.359059] ACPI: Added _OSI(Processor Device) [ 0.361024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.363025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.369081] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.375522] ACPI: Interpreter enabled [ 0.377092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.380017] ACPI: Using IOAPIC for interrupt routing [ 0.382173] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.386446] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.396813] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.399063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.403030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.407082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.412856] acpiphp: Slot [2] registered [ 0.414196] acpiphp: Slot [5] registered [ 0.416157] acpiphp: Slot [6] registered [ 0.418150] acpiphp: Slot [7] registered [ 0.419158] acpiphp: Slot [8] registered [ 0.421169] acpiphp: Slot [9] registered [ 0.423148] acpiphp: Slot [10] registered [ 0.425164] acpiphp: Slot [3] registered [ 0.426164] acpiphp: Slot [4] registered [ 0.428149] acpiphp: Slot [11] registered [ 0.430121] acpiphp: Slot [12] registered [ 0.432249] acpiphp: Slot [13] registered [ 0.433124] acpiphp: Slot [14] registered [ 0.435118] acpiphp: Slot [15] registered [ 0.437123] acpiphp: Slot [16] registered [ 0.439145] acpiphp: Slot [17] registered [ 0.441144] acpiphp: Slot [18] registered [ 0.442114] acpiphp: Slot [19] registered [ 0.444146] acpiphp: Slot [20] registered [ 0.446156] acpiphp: Slot [21] registered [ 0.448161] acpiphp: Slot [22] registered [ 0.450151] acpiphp: Slot [23] registered [ 0.451170] acpiphp: Slot [24] registered [ 0.453131] acpiphp: Slot [25] registered [ 0.455144] acpiphp: Slot [26] registered [ 0.456144] acpiphp: Slot [27] registered [ 0.458168] acpiphp: Slot [28] registered [ 0.460108] acpiphp: Slot [29] registered [ 0.462132] acpiphp: Slot [30] registered [ 0.464150] acpiphp: Slot [31] registered [ 0.466072] PCI host bridge to bus 0000:00 [ 0.468022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.470025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.473024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.476024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.479025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.482032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.485185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.488073] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.491424] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.502905] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.508067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.511039] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.514027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.518028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.522664] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.525965] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.529049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.532810] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.539018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.552015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.558025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.565459] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.577022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.585118] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.608025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.620270] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.635016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.644017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.670020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.686161] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.691021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.698023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.716022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.728612] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.741020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.748029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.764017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.774484] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.782021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.789022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.804023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.814043] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.822022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.830022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.846021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.856711] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.859467] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.863384] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.865394] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.868265] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.873036] iommu: Default domain type: Passthrough [ 0.875614] SCSI subsystem initialized [ 0.877148] ACPI: bus type USB registered [ 0.878114] usbcore: registered new interface driver usbfs [ 0.880168] usbcore: registered new interface driver hub [ 0.881074] usbcore: registered new device driver usb [ 0.882268] pps_core: LinuxPPS API ver. 1 registered [ 0.884013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.886071] PTP clock support registered [ 0.888164] EDAC MC: Ver: 3.0.0 [ 0.889523] PCI: Using ACPI for IRQ routing [ 0.890709] NetLabel: Initializing [ 0.892019] NetLabel: domain hash size = 128 [ 0.894015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.896108] NetLabel: unlabeled traffic allowed by default [ 0.898218] vgaarb: loaded [ 0.900390] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.901011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.910000] clocksource: Switched to clocksource kvm-clock [ 1.020569] VFS: Disk quotas dquot_6.6.0 [ 1.022327] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.025125] *** VALIDATE ramfs *** [ 1.026971] *** VALIDATE hugetlbfs *** [ 1.029439] pnp: PnP ACPI init [ 1.032075] pnp: PnP ACPI: found 6 devices [ 1.081626] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.086326] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.088857] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.092309] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.095091] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.098088] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.101768] NET: Registered protocol family 2 [ 1.104850] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.110397] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.114929] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.120085] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.124450] TCP: Hash tables configured (established 65536 bind 65536) [ 1.128079] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.131652] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.135424] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.138772] NET: Registered protocol family 1 [ 1.142258] RPC: Registered named UNIX socket transport module. [ 1.144695] RPC: Registered udp transport module. [ 1.146566] RPC: Registered tcp transport module. [ 1.148479] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.150786] NET: Registered protocol family 44 [ 1.152604] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.155295] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.157357] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.159439] PCI: CLS 0 bytes, default 64 [ 1.161692] Unpacking initramfs... [ 2.583404] debug: unmapping init [mem 0xffff90787cc54000-0xffff90787ffbffff] [ 2.587863] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.590665] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.594168] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.102107] Initialise system trusted keyrings [ 3.103980] Key type blacklist registered [ 3.105811] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.114883] zbud: loaded [ 3.117809] *** VALIDATE nfs *** [ 3.119099] *** VALIDATE nfs4 *** [ 3.120806] pstore: using deflate compression [ 3.124255] Platform Keyring initialized [ 3.227807] NET: Registered protocol family 38 [ 3.229955] Key type asymmetric registered [ 3.231015] Asymmetric key parser 'x509' registered [ 3.233253] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.236618] io scheduler mq-deadline registered [ 3.237729] io scheduler kyber registered [ 3.238853] io scheduler bfq registered [ 3.241223] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.244449] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.247428] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.249783] ACPI: Power Button [PWRF] [ 3.254917] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.261692] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.274682] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.281811] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.296567] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.325068] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.356065] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.361538] Non-volatile memory driver v1.3 [ 3.363357] Linux agpgart interface v0.103 [ 3.394398] virtio_blk virtio1: [vda] 147232 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.397810] vda: detected capacity change from 0 to 75382784 [ 3.415782] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.419197] vdb: detected capacity change from 0 to 1073741824 [ 3.437054] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.440508] vdc: detected capacity change from 0 to 2621440000 [ 3.454612] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.457622] vdd: detected capacity change from 0 to 2621440000 [ 3.472487] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.474766] vde: detected capacity change from 0 to 4294967296 [ 3.489686] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.493204] vdf: detected capacity change from 0 to 4294967296 [ 3.501294] libphy: Fixed MDIO Bus: probed [ 3.507758] usbcore: registered new interface driver usbserial_generic [ 3.510580] usbserial: USB Serial support registered for generic [ 3.513785] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.518733] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.520504] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.523308] mousedev: PS/2 mouse device common for all mice [ 3.526261] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.526589] rtc_cmos 00:05: RTC can wake from S4 [ 3.533889] rtc_cmos 00:05: registered as rtc0 [ 3.536709] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.537247] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.544314] intel_pstate: CPU model not supported [ 3.546797] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.549989] hid: raw HID events driver (C) Jiri Kosina [ 3.552594] usbcore: registered new interface driver usbhid [ 3.554607] usbhid: USB HID core driver [ 3.556245] drop_monitor: Initializing network drop monitor service [ 3.558752] Initializing XFRM netlink socket [ 3.560598] NET: Registered protocol family 10 [ 3.563370] Segment Routing with IPv6 [ 3.565069] NET: Registered protocol family 17 [ 3.568050] mpls_gso: MPLS GSO support [ 3.574358] RAS: Correctable Errors collector initialized. [ 3.577112] AVX version of gcm_enc/dec engaged. [ 3.579262] AES CTR mode by8 optimization enabled [ 3.659470] sched_clock: Marking stable (3659447915, 0)->(4587350693, -927902778) [ 3.662987] registered taskstats version 1 [ 3.665174] Loading compiled-in X.509 certificates [ 3.667196] zswap: loaded using pool lzo/zbud [ 3.690598] Key type big_key registered [ 3.703962] Key type encrypted registered [ 3.705918] ima: No TPM chip found, activating TPM-bypass! [ 3.708178] ima: Allocated hash algorithm: sha1 [ 3.710353] ima: No architecture policies found [ 3.712180] evm: Initialising EVM extended attributes: [ 3.714456] evm: security.selinux [ 3.715742] evm: security.ima [ 3.716763] evm: security.capability [ 3.718074] evm: HMAC attrs: 0x1 [ 3.720692] rtc_cmos 00:05: setting system clock to 2026-09-11 08:29:35 UTC (1789115375) [ 3.727316] debug: unmapping init [mem 0xffffffffb4203000-0xffffffffb43fffff] [ 3.730364] debug: unmapping init [mem 0xffffffffb2f82000-0xffffffffb3258fff] [ 3.739069] Write protecting the kernel read-only data: 28672k [ 3.742763] debug: unmapping init [mem 0xffffffffb1603000-0xffffffffb17fffff] [ 3.745713] debug: unmapping init [mem 0xffffffffb1f14000-0xffffffffb1ffffff] [ 3.779288] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.787603] systemd[1]: Detected virtualization kvm. [ 3.789658] systemd[1]: Detected architecture x86-64. [ 3.791812] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.818697] systemd[1]: No hostname configured. [ 3.820832] systemd[1]: Set hostname to . [ 3.822985] random: systemd: uninitialized urandom read (16 bytes read) [ 3.825745] systemd[1]: Initializing machine ID from random generator. [ 3.968117] random: systemd: uninitialized urandom read (16 bytes read) [ 3.970721] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.977314] random: systemd: uninitialized urandom read (16 bytes read) [ 3.980290] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.984473] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.619762] device-mapper: uevent: version 1.0.3 [ 4.622094] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.388540] virtio_net virtio0 ens2: renamed from eth0 [ 5.393005] random: fast init done [ 5.416933] scsi host0: ata_piix [ 5.436521] scsi host1: ata_piix [ 5.438400] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.440469] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.239561] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.222755] random: crng init done [ 10.225629] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.702574] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.787353] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.018642] SELinux: Disabled at runtime. [ 12.081097] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.091105] systemd[1]: Detected virtualization kvm. [ 12.093489] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.548879] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.553277] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.558354] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.564304] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.567969] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.573865] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.579754] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on[ 12.705509] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.055915] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.356739] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.361337] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.470562] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.489989] EDAC sbridge: Ver: 1.1.2 [ 15.084815] Key type dns_resolver registered [ 15.387458] NFS: Registering the id_resolver key type [ 15.389299] Key type id_resolver registered [ 15.390836] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg304-server login: [ 49.377388] libcfs: loading out-of-tree module taints kernel. [ 49.488933] Key type ._llcrypt registered [ 49.498311] Key type .llcrypt registered [ 49.644938] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 70.360301] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 72.269635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.311944] alg: No test for adler32 (adler32-zlib) [ 73.589378] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 74.159463] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 75.855760] Key type lgssc registered [ 77.612192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.374303] hrtimer: interrupt took 2789819 ns [ 95.917613] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 136.788836] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 148.638913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 148.673737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.009634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.039846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.173330] Lustre: lustre-MDT0000: new disk, initializing [ 150.250716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.278558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 154.912769] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.602062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.675675] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.716778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.721450] Lustre: Skipped 1 previous similar message [ 167.794855] Lustre: lustre-MDT0001: new disk, initializing [ 167.867221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.921047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 167.937910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 170.963844] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.195792] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 183.570447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 183.810387] Lustre: lustre-OST0000: new disk, initializing [ 183.813771] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 183.819605] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 183.878340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.874732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 187.887198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 187.951466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 190.056714] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.660218] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 204.864886] Lustre: lustre-OST0001: new disk, initializing [ 204.872071] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 204.883531] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 204.963156] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.936028] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.053716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 214.078108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.122787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 224.676860] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 231.819946] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 238.267459] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing check_logdir /tmp/testlogs/ [ 243.544921] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing yml_node [ 248.158566] Lustre: DEBUG MARKER: Client: 2.17.57.142 [ 250.774644] Lustre: DEBUG MARKER: MDS: 2.17.57.142 [ 253.425959] Lustre: DEBUG MARKER: OSS: 2.17.57.142 [ 254.965629] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 04:33:45 EDT 2026 [ 273.468866] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 275.109612] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.785142] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.789398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.794906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 290.806605] Lustre: Skipped 3 previous similar messages [ 290.812099] Lustre: Skipped 2 previous similar messages [ 295.924155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 295.927035] Lustre: Skipped 1 previous similar message [ 296.926362] Lustre: server umount lustre-MDT0000 complete [ 300.943936] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789115672 with bad export cookie 15498293266839195770 [ 300.947397] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 300.952635] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 301.027572] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.034966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.052871] Lustre: Skipped 1 previous similar message [ 305.056438] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 305.062921] Lustre: Skipped 2 previous similar messages [ 310.243962] LustreError: 6516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.245252] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.270732] LustreError: 6516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 310.271728] Lustre: Skipped 1 previous similar message [ 311.266902] LustreError: 6516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.297502] LustreError: 6516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 315.359134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 315.632951] Lustre: server umount lustre-MDT0001 complete [ 319.403231] Lustre: server umount lustre-OST0000 complete [ 323.539900] Lustre: server umount lustre-OST0001 complete [ 332.932304] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 344.062992] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 382.643295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 383.005308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 383.029100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 383.147593] Lustre: lustre-MDT0000: new disk, initializing [ 383.276666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 383.321030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 388.567706] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 399.616550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.742868] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 399.789967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 399.793626] Lustre: Skipped 1 previous similar message [ 399.861875] Lustre: lustre-MDT0001: new disk, initializing [ 400.004481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 400.057123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 400.067673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 404.626186] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 414.298442] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 420.308732] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 426.734988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 426.992337] Lustre: lustre-OST0000: new disk, initializing [ 426.995626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 427.004470] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 427.087573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.059203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 429.070814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 429.187690] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 432.701912] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 441.340733] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 445.064194] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 445.335031] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.639226] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.847907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 454.629804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 454.644175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.658660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 454.664695] Lustre: Skipped 1 previous similar message [ 454.677849] Lustre: Skipped 3 previous similar messages [ 457.772358] Lustre: server umount lustre-OST0000 complete [ 464.864643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.880632] Lustre: Skipped 1 previous similar message [ 467.875606] Lustre: server umount lustre-MDT0000 complete [ 469.985535] LustreError: 17479:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 470.007849] LustreError: 17479:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 471.407071] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789115843 with bad export cookie 15498293266839197303 [ 471.414200] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 471.423476] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 471.819824] Lustre: server umount lustre-MDT0001 complete [ 475.482977] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:37:25 (1789115845) [ 481.796662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.228877] LustreError: 19306:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.287434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 486.689702] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 487.408189] LustreError: 19307:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.060421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.628364] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 508.779708] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 515.080583] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 522.739535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 529.106240] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 536.967522] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 547.815238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 547.822231] LustreError: Skipped 1 previous similar message [ 547.825579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.845648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 547.849037] Lustre: Skipped 2 previous similar messages [ 550.884263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.666313] Lustre: server umount lustre-OST0000 complete [ 561.121547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.134117] Lustre: Skipped 2 previous similar messages [ 563.482484] Lustre: server umount lustre-MDT0000 complete [ 566.245318] LustreError: 20034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.274334] LustreError: 20034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 567.278823] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789115939 with bad export cookie 15498293266839198185 [ 567.283268] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 567.291436] LustreError: 19288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 567.669404] Lustre: server umount lustre-MDT0001 complete [ 580.813043] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 583.923527] Key type lgssc unregistered [ 584.334985] LNet: 22846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.342725] LNetError: 22846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.359100] LNet: Removed LNI 192.168.203.104@tcp [ 585.533689] Key type .llcrypt unregistered [ 585.537696] Key type ._llcrypt unregistered [ 610.314846] Key type ._llcrypt registered [ 610.322337] Key type .llcrypt registered [ 610.487693] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 612.375892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 612.421465] alg: No test for adler32 (adler32-zlib) [ 613.771097] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 614.127412] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 615.863475] Key type lgssc registered [ 617.709146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 626.991730] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 627.014520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.503407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.305588] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 638.945519] LustreError: 24264:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.977031] LustreError: 24264:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 641.363950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.784664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 645.758128] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 653.931571] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 660.694094] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 668.820190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.345251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 673.460430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 676.245316] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 684.371450] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 702.432857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 702.442580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.487423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 702.493934] Lustre: Skipped 1 previous similar message [ 707.492539] Lustre: server umount lustre-OST0000 complete [ 712.680661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.693135] Lustre: Skipped 2 previous similar messages [ 712.694828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.706922] Lustre: Skipped 1 previous similar message [ 717.624632] Lustre: server umount lustre-MDT0000 complete [ 717.801610] LustreError: 24264:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 717.815716] LustreError: 24264:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 721.404265] LustreError: 24246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116093 with bad export cookie 10646912058641407395 [ 721.406428] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.417880] LustreError: 24246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 721.837786] Lustre: server umount lustre-MDT0001 complete [ 732.444066] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 735.354541] Key type lgssc unregistered [ 735.702736] LNet: 28247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.709078] LNetError: 28247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.726761] LNet: Removed LNI 192.168.203.104@tcp [ 736.520385] Key type .llcrypt unregistered [ 736.523767] Key type ._llcrypt unregistered [ 760.003038] Key type ._llcrypt registered [ 760.004665] Key type .llcrypt registered [ 760.130741] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 761.109612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 761.175585] alg: No test for adler32 (adler32-zlib) [ 762.271956] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 762.487214] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 764.167214] Key type lgssc registered [ 765.446240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 773.541271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 773.582691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.119928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 779.813603] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 784.362835] LustreError: 29661:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 784.382314] LustreError: 29661:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 789.153772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.477207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 789.744551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 795.372659] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 803.666724] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 809.699660] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 824.452670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 824.771582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.589518] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 839.594354] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 846.260876] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 856.544654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 856.551364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.562568] Lustre: Skipped 1 previous similar message [ 856.562884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 856.582134] Lustre: Skipped 1 previous similar message [ 861.144392] Lustre: server umount lustre-OST0000 complete [ 866.788446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.790013] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 866.792615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 866.808150] Lustre: Skipped 1 previous similar message [ 866.819049] LustreError: Skipped 1 previous similar message [ 870.941878] Lustre: server umount lustre-MDT0000 complete [ 871.907826] LustreError: 29661:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 871.920460] LustreError: 29661:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 874.270059] LustreError: 31849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116246 with bad export cookie 2429275752462800415 [ 874.281978] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.552483] Lustre: server umount lustre-MDT0001 complete [ 884.351796] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 887.429261] Key type lgssc unregistered [ 887.713070] LNet: 33564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.717825] LNetError: 33564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.732983] LNet: Removed LNI 192.168.203.104@tcp [ 888.595510] Key type .llcrypt unregistered [ 888.604020] Key type ._llcrypt unregistered [ 912.973654] Key type ._llcrypt registered [ 912.975527] Key type .llcrypt registered [ 913.072810] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 914.308452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 914.337503] alg: No test for adler32 (adler32-zlib) [ 915.372177] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 915.578515] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 917.247433] Key type lgssc registered [ 918.575564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 926.346832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 926.378574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.916397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 932.485590] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 937.443405] LustreError: 34981:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.459235] LustreError: 34981:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 941.892879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.318532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 946.679185] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 955.079758] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 961.625400] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 970.313473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 970.783294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.854612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 976.246452] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 984.922558] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 998.371697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.373950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 998.377182] Lustre: Skipped 1 previous similar message [ 998.379526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 998.380513] LustreError: Skipped 1 previous similar message [ 1003.494335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.505134] Lustre: Skipped 2 previous similar messages [ 1004.381450] Lustre: server umount lustre-OST0000 complete [ 1008.608651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.621899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.632152] Lustre: Skipped 1 previous similar message [ 1008.643914] Lustre: Skipped 1 previous similar message [ 1013.729732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1013.740475] Lustre: Skipped 1 previous similar message [ 1014.585063] Lustre: server umount lustre-MDT0000 complete [ 1018.168150] LustreError: 34963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116389 with bad export cookie 5197899404137118159 [ 1018.170649] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1018.183256] LustreError: 34963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1018.525789] Lustre: server umount lustre-MDT0001 complete [ 1030.016680] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1032.986980] Key type lgssc unregistered [ 1033.282556] LNet: 38539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.291398] LNetError: 38539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.309222] LNet: Removed LNI 192.168.203.104@tcp [ 1034.069827] Key type .llcrypt unregistered [ 1034.073596] Key type ._llcrypt unregistered [ 1057.528254] Key type ._llcrypt registered [ 1057.531662] Key type .llcrypt registered [ 1057.644934] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1058.840955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1058.906177] alg: No test for adler32 (adler32-zlib) [ 1059.967939] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 1060.259443] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1061.975164] Key type lgssc registered [ 1063.227649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1070.464713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1070.497495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1072.035425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.642602] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1081.322382] LustreError: 39956:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.346353] LustreError: 39956:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1085.682588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.082500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1090.571556] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1098.725232] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1104.226641] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1110.562978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1110.920611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1117.130953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1117.403548] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1126.041178] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1134.089489] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1137.638748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.644268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1137.658735] Lustre: Skipped 1 previous similar message [ 1137.659161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1137.672253] Lustre: Skipped 1 previous similar message [ 1139.196508] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1142.762612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1142.788094] Lustre: Skipped 1 previous similar message [ 1147.359158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1147.606559] Lustre: server umount lustre-OST0000 complete [ 1167.840246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1167.851097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.874262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1167.883337] Lustre: Skipped 1 previous similar message [ 1172.456185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.980676] Lustre: server umount lustre-MDT0000 complete [ 1177.568141] LustreError: 41196:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.583287] LustreError: 41196:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1177.850453] LustreError: 39937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116549 with bad export cookie 2160886927409461295 [ 1177.853443] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1177.860946] LustreError: 39937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1184.228797] Lustre: server umount lustre-MDT0001 complete [ 1194.748072] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1197.145341] Key type lgssc unregistered [ 1197.372811] LNet: 43671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.384234] LNetError: 43671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.407552] LNet: Removed LNI 192.168.203.104@tcp [ 1198.096622] Key type .llcrypt unregistered [ 1198.099394] Key type ._llcrypt unregistered [ 1218.804721] Key type ._llcrypt registered [ 1218.820383] Key type .llcrypt registered [ 1218.938083] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1219.527689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1219.568696] alg: No test for adler32 (adler32-zlib) [ 1220.576975] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 1220.752719] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1222.417187] Key type lgssc registered [ 1223.467980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1230.269192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1230.319436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.741211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.966365] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1242.082132] LustreError: 45063:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1242.099729] LustreError: 45063:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1247.204518] LustreError: 45064:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.472225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.002536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1252.858460] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1261.943551] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1267.862729] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1276.326681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1276.849793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.894925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1284.179257] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1293.533201] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1303.521623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.534817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1303.538204] Lustre: Skipped 2 previous similar messages [ 1303.545009] Lustre: Skipped 1 previous similar message [ 1306.618916] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1308.648470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1308.658715] Lustre: Skipped 2 previous similar messages [ 1311.718466] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 1316.319212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1316.611672] Lustre: server umount lustre-MDT0000 complete [ 1316.843180] LustreError: 46068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1318.894142] LustreError: 46069:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1318.910441] LustreError: 46069:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1320.542916] LustreError: 45044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789116692 with bad export cookie 12568955610080706588 [ 1320.546277] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1320.554707] LustreError: 45044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1321.978469] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 1321.982629] Lustre: Skipped 3 previous similar messages [ 1324.000615] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.004477] LustreError: 45058:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.030088] LustreError: 45058:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1327.051852] Lustre: server umount lustre-MDT0001 complete [ 1343.437686] Lustre: server umount lustre-OST0000 complete [ 1360.901235] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1364.196985] Key type lgssc unregistered [ 1364.543645] LNet: 48927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.553857] LNetError: 48927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.625866] LNet: Removed LNI 192.168.203.104@tcp [ 1366.541667] Key type .llcrypt unregistered [ 1366.544207] Key type ._llcrypt unregistered [ 1392.294466] Key type ._llcrypt registered [ 1392.313893] Key type .llcrypt registered [ 1392.502886] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1393.422377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1393.555291] alg: No test for adler32 (adler32-zlib) [ 1394.715968] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 1395.111056] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1396.855178] Key type lgssc registered [ 1398.722497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1408.970158] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1408.989418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1424.543602] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1429.663311] LustreError: 50318:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 1456.351269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.505109] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1469.034256] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1635.256293] Lustre: server umount lustre-OST0000 complete [ 1655.174705] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1658.720522] Key type lgssc unregistered [ 1659.094629] LNet: 52003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.105945] LNetError: 52003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.125390] LNet: Removed LNI 192.168.203.104@tcp [ 1659.877605] Key type .llcrypt unregistered [ 1659.880685] Key type ._llcrypt unregistered [ 1683.413623] Key type ._llcrypt registered [ 1683.417248] Key type .llcrypt registered [ 1683.498972] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1684.517728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1684.631089] alg: No test for adler32 (adler32-zlib) [ 1685.789840] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 1685.992605] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1687.663178] Key type lgssc registered [ 1689.022707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1698.196226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1698.215600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.738709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1705.210798] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1710.050658] LustreError: 53418:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1710.089945] LustreError: 53418:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1711.074615] LustreError: 53417:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1714.448731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1714.699912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1719.498581] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1726.973686] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1732.046592] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1738.335962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1738.731261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1745.341425] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1745.909470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1753.215622] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1756.128540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1756.140385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1756.180940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.181946] Lustre: Skipped 1 previous similar message [ 1756.205070] Lustre: Skipped 1 previous similar message [ 1761.254680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1762.213654] Lustre: server umount lustre-OST0000 complete [ 1766.370806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.393097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1766.400147] Lustre: Skipped 2 previous similar messages [ 1771.491375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1771.498945] Lustre: Skipped 1 previous similar message [ 1772.642861] Lustre: server umount lustre-MDT0000 complete [ 1776.610445] LustreError: 56177:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1776.634596] LustreError: 56177:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1778.361525] LustreError: 53399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117150 with bad export cookie 16717901998579000619 [ 1778.382897] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1778.398772] LustreError: 53399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1779.051717] Lustre: server umount lustre-MDT0001 complete [ 1794.372370] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1797.543395] Key type lgssc unregistered [ 1797.839673] LNet: 56955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.851666] LNetError: 56955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.892662] LNet: Removed LNI 192.168.203.104@tcp [ 1799.922147] Key type .llcrypt unregistered [ 1799.924368] Key type ._llcrypt unregistered [ 1822.871995] Key type ._llcrypt registered [ 1822.876637] Key type .llcrypt registered [ 1822.965883] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1823.988846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.014641] alg: No test for adler32 (adler32-zlib) [ 1825.115488] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 1825.384291] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1827.039174] Key type lgssc registered [ 1828.026214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1836.002417] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1836.020624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1851.679926] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1856.800489] LustreError: 58343:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 1883.488125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.129400] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1898.848901] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1906.244565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.744625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.816547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1911.778137] LustreError: 58368:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1911.780543] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1911.793689] LustreError: 58368:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1916.903979] LustreError: 58369:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.930588] LustreError: 58368:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1920.658936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.076598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1925.753537] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1933.247773] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1938.868655] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1941.477552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.480966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1941.482721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1941.496348] Lustre: Skipped 1 previous similar message [ 1941.514893] Lustre: Skipped 1 previous similar message [ 1946.602382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1946.606231] Lustre: Skipped 1 previous similar message [ 1951.714291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1951.717763] Lustre: Skipped 1 previous similar message [ 1955.295118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1955.396611] Lustre: server umount lustre-OST0000 complete [ 1977.315019] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1977.331497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1977.336532] Lustre: Skipped 1 previous similar message [ 1981.409599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1981.416082] Lustre: Skipped 1 previous similar message [ 1982.349955] Lustre: server umount lustre-MDT0000 complete [ 1986.167860] LustreError: 58350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117357 with bad export cookie 4439271188008249641 [ 1986.169423] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1986.179431] LustreError: 58350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1992.580870] Lustre: server umount lustre-MDT0001 complete [ 2003.056823] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2005.639031] Key type lgssc unregistered [ 2005.945254] LNet: 62105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.954298] LNetError: 62105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.969595] LNet: Removed LNI 192.168.203.104@tcp [ 2006.703111] Key type .llcrypt unregistered [ 2006.704879] Key type ._llcrypt unregistered [ 2030.682735] Key type ._llcrypt registered [ 2030.683910] Key type .llcrypt registered [ 2030.833502] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2032.366850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2032.412593] alg: No test for adler32 (adler32-zlib) [ 2033.539439] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 2033.824123] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2035.491467] Key type lgssc registered [ 2037.370460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2046.115942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2046.158577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.691519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.672947] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2058.221156] LustreError: 63521:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.241744] LustreError: 63521:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2060.388743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.692212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2065.523493] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2073.957565] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2079.782365] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2085.718985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2086.087290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.897149] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2094.253657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2098.692918] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2120.161706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2120.183027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.197039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2121.184601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2121.190818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2121.199902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.677466] Lustre: server umount lustre-OST0000 complete [ 2131.428525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.441454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.461354] Lustre: Skipped 1 previous similar message [ 2131.518304] Lustre: Skipped 2 previous similar messages [ 2134.683129] Lustre: server umount lustre-MDT0000 complete [ 2136.545749] LustreError: 64720:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.574330] LustreError: 64720:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2138.265038] LustreError: 63501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117510 with bad export cookie 18138090514720044209 [ 2138.266536] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2138.273842] LustreError: 63501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2138.668880] Lustre: server umount lustre-MDT0001 complete [ 2150.687374] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2153.907162] Key type lgssc unregistered [ 2154.197763] LNet: 67079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.203321] LNetError: 67079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.217895] LNet: Removed LNI 192.168.203.104@tcp [ 2155.071946] Key type .llcrypt unregistered [ 2155.074983] Key type ._llcrypt unregistered [ 2180.978676] Key type ._llcrypt registered [ 2180.980849] Key type .llcrypt registered [ 2181.095864] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2182.252684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.310111] alg: No test for adler32 (adler32-zlib) [ 2183.430338] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 2183.788606] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2185.527127] Key type lgssc registered [ 2187.282831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2199.256796] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 05:06:09 (1789117569) [ 2205.950644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2206.004098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.566647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.192700] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2217.959787] LustreError: 68784:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.987180] LustreError: 68784:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2220.497900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.009461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2226.131785] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2233.906817] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2239.353066] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2245.851508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2246.231084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.885442] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2252.404286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2258.894566] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2268.796944] Lustre: Failing over lustre-MDT0000 [ 2269.027727] Lustre: server umount lustre-MDT0000 complete [ 2271.203517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.208450] LustreError: 68780:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2271.220253] Lustre: Skipped 2 previous similar messages [ 2271.231177] LustreError: 68780:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2276.326272] LustreError: 68779:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.338514] LustreError: 68779:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2277.651217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.798694] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2277.942814] Lustre: *** cfs_fail_loc=135, val=0*** [ 2278.053974] LustreError: 71870:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2278.070072] LustreError: 71870:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -2 [ 2278.079671] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2278.079671] [ 2278.087661] LustreError: MGC192.168.203.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2278.109808] LustreError: 71858:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2278.115589] LustreError: 71858:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2278.121857] LustreError: 71858:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2278.129278] LustreError: 71858:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2278.166968] Lustre: server umount lustre-MDT0000 complete [ 2278.168478] LustreError: 71858:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2281.443712] LustreError: 69511:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.468940] LustreError: 69511:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2284.312042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.400148] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2284.605749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.652743] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2287.083869] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2288.628397] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2289.646859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2289.694053] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2289.754167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2296.803132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2296.808491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2296.825841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.879803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2299.885681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.586213] Lustre: server umount lustre-OST0000 complete [ 2304.995536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2304.995783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2305.016575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2305.020728] Lustre: Skipped 1 previous similar message [ 2310.120282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2310.123868] Lustre: Skipped 2 previous similar messages [ 2310.371908] Lustre: server umount lustre-MDT0000 complete [ 2313.835929] LustreError: 68764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117685 with bad export cookie 1587656235423755814 [ 2313.842071] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2313.848779] LustreError: 68764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2314.232143] Lustre: server umount lustre-MDT0001 complete [ 2326.046381] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2329.809316] Key type lgssc unregistered [ 2330.254093] LNet: 73653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2330.269348] LNetError: 73653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.307578] LNet: Removed LNI 192.168.203.104@tcp [ 2332.092826] Key type .llcrypt unregistered [ 2332.096337] Key type ._llcrypt unregistered [ 2353.296076] Key type ._llcrypt registered [ 2353.297958] Key type .llcrypt registered [ 2353.403745] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2354.090506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2354.177263] alg: No test for adler32 (adler32-zlib) [ 2355.154602] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 2355.384254] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2357.047202] Key type lgssc registered [ 2357.998518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2365.083881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2365.096455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.401891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.874596] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2376.682623] LustreError: 75068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.710684] LustreError: 75068:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2378.960608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.301221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2383.432417] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2391.370401] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2397.380209] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2405.005562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2405.437428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.584719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2412.239963] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2421.144428] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2429.899448] Lustre: Failing over lustre-MDT0000 [ 2429.933478] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.940775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.946432] Lustre: Skipped 2 previous similar messages [ 2429.957094] Lustre: Skipped 1 previous similar message [ 2430.238853] Lustre: server umount lustre-MDT0000 complete [ 2433.027180] LustreError: 75064:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.045242] LustreError: 75064:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.078015] LustreError: 75064:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2438.116806] LustreError: 75063:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2439.930345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2440.035335] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.133945] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2440.166887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2440.180125] Lustre: Skipped 2 previous similar messages [ 2440.255594] LustreError: 78154:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2440.266728] LustreError: 78154:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -9 [ 2440.277581] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2440.277581] [ 2440.288064] LustreError: MGC192.168.203.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2440.295340] LustreError: 78143:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2440.301497] LustreError: 78143:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2440.314585] LustreError: 78143:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2440.321399] LustreError: 78143:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2440.346905] Lustre: server umount lustre-MDT0000 complete [ 2440.352529] LustreError: 78143:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2443.246743] LustreError: 75065:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.434681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.609200] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2447.919942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.983363] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2448.366630] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2452.896585] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2452.980832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2453.011274] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2453.081333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2463.203939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.213883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2463.224592] Lustre: Skipped 1 previous similar message [ 2463.248401] Lustre: Skipped 1 previous similar message [ 2467.750978] Lustre: server umount lustre-OST0000 complete [ 2473.449749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2473.451677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.463601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.466480] Lustre: Skipped 1 previous similar message [ 2477.964530] Lustre: server umount lustre-MDT0000 complete [ 2478.565485] LustreError: 75068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.597518] LustreError: 75068:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2481.857807] LustreError: 75049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789117853 with bad export cookie 12747920836674653361 [ 2481.867919] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2482.276144] Lustre: server umount lustre-MDT0001 complete [ 2494.448664] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2497.068303] Key type lgssc unregistered [ 2497.326314] LNet: 79940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2497.335405] LNetError: 79940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2497.363305] LNet: Removed LNI 192.168.203.104@tcp [ 2498.412207] Key type .llcrypt unregistered [ 2498.414078] Key type ._llcrypt unregistered [ 2513.306762] Key type ._llcrypt registered [ 2513.310206] Key type .llcrypt registered [ 2513.441245] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2513.885329] Key type .llcrypt unregistered [ 2513.887063] Key type ._llcrypt unregistered [ 2537.827262] Key type ._llcrypt registered [ 2537.829231] Key type .llcrypt registered [ 2537.957593] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2539.175516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2539.270032] alg: No test for adler32 (adler32-zlib) [ 2540.502831] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 2540.761541] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2542.447873] Key type lgssc registered [ 2543.780708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2552.573747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2552.607610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.083109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.841762] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2563.556225] LustreError: 81846:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.574615] LustreError: 81846:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2567.189123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.594064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2571.671683] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2579.491772] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2585.072975] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2591.937401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2592.355921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.333528] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2598.537435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2605.360892] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2614.246368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2614.248068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.254656] LustreError: Skipped 1 previous similar message [ 2614.257454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2614.266853] Lustre: Skipped 1 previous similar message [ 2614.281029] Lustre: Skipped 1 previous similar message [ 2619.364850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2619.371220] Lustre: Skipped 1 previous similar message [ 2620.075501] Lustre: server umount lustre-OST0000 complete [ 2623.971419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2623.980418] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2623.992592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.602554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2629.612652] Lustre: Skipped 3 previous similar messages [ 2629.836201] Lustre: server umount lustre-MDT0000 complete [ 2633.328293] LustreError: 81827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118005 with bad export cookie 79186859869770491 [ 2633.332763] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2633.344146] LustreError: 81827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2633.687672] Lustre: server umount lustre-MDT0001 complete [ 2643.192478] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2645.471160] Key type lgssc unregistered [ 2645.823966] LNet: 85385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2645.837899] LNetError: 85385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2645.869796] LNet: Removed LNI 192.168.203.104@tcp [ 2646.695062] Key type .llcrypt unregistered [ 2646.698116] Key type ._llcrypt unregistered [ 2673.189863] Key type ._llcrypt registered [ 2673.207039] Key type .llcrypt registered [ 2673.337353] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2674.839171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2674.903457] alg: No test for adler32 (adler32-zlib) [ 2675.915992] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 2676.122969] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2677.831272] Key type lgssc registered [ 2679.498731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2687.638991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2687.658677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.224677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.299755] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2698.754194] LustreError: 86800:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.778711] LustreError: 86800:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2701.665837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.989594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2706.590180] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2713.787979] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2719.330462] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2726.723686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2726.909811] LustreError: 88622:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.136225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.136240] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2733.630133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2741.659420] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2748.903954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2748.907245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2748.908887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2748.917899] Lustre: Skipped 1 previous similar message [ 2750.712810] Lustre: server umount lustre-OST0000 complete [ 2758.625836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2758.634523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.648033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2758.659124] Lustre: Skipped 1 previous similar message [ 2760.678826] Lustre: server umount lustre-MDT0000 complete [ 2764.260023] LustreError: 87816:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.271820] LustreError: 87816:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2764.361467] LustreError: 89296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118136 with bad export cookie 17904902835579886537 [ 2764.363335] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.381711] LustreError: 89296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2764.771507] Lustre: server umount lustre-MDT0001 complete [ 2774.647230] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2777.365794] Key type lgssc unregistered [ 2777.656749] LNet: 90338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2777.670832] LNetError: 90338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2777.687622] LNet: Removed LNI 192.168.203.104@tcp [ 2778.610505] Key type .llcrypt unregistered [ 2778.612218] Key type ._llcrypt unregistered [ 2802.611831] Key type ._llcrypt registered [ 2802.614323] Key type .llcrypt registered [ 2802.730378] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2803.896223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2804.038585] alg: No test for adler32 (adler32-zlib) [ 2805.165474] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 2805.424356] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2807.106687] Key type lgssc registered [ 2808.300380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2816.346979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2816.360503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.837464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2822.444784] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2827.246565] LustreError: 91751:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.273413] LustreError: 91751:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2831.254092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2831.604262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2835.829789] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2843.503150] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2849.838804] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2855.932263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.221775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.764320] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2862.407331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2869.617642] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2888.166714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.171384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2888.181385] Lustre: Skipped 1 previous similar message [ 2888.182096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2888.188510] Lustre: Skipped 1 previous similar message [ 2889.977326] Lustre: server umount lustre-OST0000 complete [ 2893.291686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.292599] LustreError: 91059:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cb427963 x1876026381597184/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2893.298514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2893.301371] Lustre: Skipped 1 previous similar message [ 2893.332346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.337853] Lustre: Skipped 1 previous similar message [ 2898.403574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2898.422071] Lustre: Skipped 1 previous similar message [ 2899.571616] Lustre: server umount lustre-MDT0000 complete [ 2902.867601] LustreError: 94246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118274 with bad export cookie 14264304427969092740 [ 2902.868710] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2902.883078] LustreError: 94246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2903.265245] Lustre: server umount lustre-MDT0001 complete [ 2912.665493] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2915.610757] Key type lgssc unregistered [ 2915.846929] LNet: 95311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2915.863545] LNetError: 95311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2915.873982] LNet: Removed LNI 192.168.203.104@tcp [ 2916.682414] Key type .llcrypt unregistered [ 2916.687382] Key type ._llcrypt unregistered [ 2941.949671] Key type ._llcrypt registered [ 2941.953751] Key type .llcrypt registered [ 2942.120810] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2943.432674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2943.506196] alg: No test for adler32 (adler32-zlib) [ 2944.607685] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 2944.896880] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2946.575914] Key type lgssc registered [ 2948.484990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2958.628610] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2958.694562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2974.431383] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2979.552124] LustreError: 96699:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 3006.176115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.031569] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3020.042721] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3025.287255] Lustre: server umount lustre-OST0000 complete [ 3033.140735] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 05:20:03 (1789118403) [ 3039.280430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.675039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.701940] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3044.850061] LustreError: 98340:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3044.887207] LustreError: 98340:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3049.958242] LustreError: 98339:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3051.910473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.350817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3056.722863] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3062.928562] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3068.525143] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3075.495447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3075.990821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3082.357847] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3085.237916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3090.679082] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3103.715270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3103.722076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.738304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3105.779047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3105.779157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3105.783244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3108.582665] Lustre: server umount lustre-OST0000 complete [ 3113.953451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3113.961777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.974742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3113.979879] Lustre: Skipped 1 previous similar message [ 3116.001240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3116.005173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.705152] Lustre: server umount lustre-MDT0000 complete [ 3119.008820] LustreError: 98340:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.030978] LustreError: 98340:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3122.201601] LustreError: 98321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118493 with bad export cookie 15802724434484570165 [ 3122.203293] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3122.213917] LustreError: 98321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3122.547545] Lustre: server umount lustre-MDT0001 complete [ 3132.832255] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3135.340800] Key type lgssc unregistered [ 3135.661722] LNet: 102035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3135.672760] LNetError: 102035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3136.754755] LNet: Removed LNI 192.168.203.104@tcp [ 3137.805163] Key type .llcrypt unregistered [ 3137.807829] Key type ._llcrypt unregistered [ 3184.130512] Key type ._llcrypt registered [ 3184.136414] Key type .llcrypt registered [ 3184.239277] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3185.133535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3185.226268] alg: No test for adler32 (adler32-zlib) [ 3186.302844] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 3186.560485] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3188.263163] Key type lgssc registered [ 3189.316547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3197.563513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3197.612696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.167303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.012489] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3208.680201] LustreError: 104232:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.698728] LustreError: 104232:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3212.272572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3212.695271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3217.070547] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3224.344883] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3229.411947] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3235.795399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.223680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3241.622848] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3243.390596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3248.600694] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3258.855852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3258.864539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.869136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3258.880953] Lustre: Skipped 1 previous similar message [ 3258.893851] Lustre: Skipped 1 previous similar message [ 3263.495292] Lustre: server umount lustre-OST0000 complete [ 3269.089333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3269.098955] LustreError: Skipped 1 previous similar message [ 3269.101194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.105869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3269.130369] Lustre: Skipped 1 previous similar message [ 3273.475777] Lustre: server umount lustre-MDT0000 complete [ 3274.208580] LustreError: 104233:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.222193] LustreError: 104233:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3277.215895] LustreError: 104214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118648 with bad export cookie 15150754646236550926 [ 3277.227980] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3277.230018] LustreError: 104214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3277.696954] Lustre: server umount lustre-MDT0001 complete [ 3289.413830] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3292.094717] Key type lgssc unregistered [ 3292.337554] LNet: 107768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3292.344940] LNetError: 107768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3292.367599] LNet: Removed LNI 192.168.203.104@tcp [ 3293.286553] Key type .llcrypt unregistered [ 3293.289352] Key type ._llcrypt unregistered [ 3310.730715] Key type ._llcrypt registered [ 3310.735545] Key type .llcrypt registered [ 3310.832794] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3311.760267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3311.798044] alg: No test for adler32 (adler32-zlib) [ 3312.889989] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 3313.103753] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3314.767252] Key type lgssc registered [ 3315.893885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3323.405162] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3323.428654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3338.975611] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3344.095478] LustreError: 109011:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 3370.783575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.070042] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3385.122345] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3392.506697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.931606] LustreError: MGC192.168.203.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3392.940525] LustreError: 110035:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3392.946735] LustreError: 110035:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3392.952849] LustreError: 110035:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3393.027463] Lustre: server umount lustre-MDT0000 complete [ 3393.035231] LustreError: 110035:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3410.045199] Lustre: server umount lustre-OST0000 complete [ 3420.479092] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3430.577869] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3467.368206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3467.827461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3467.887500] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3468.020397] Lustre: lustre-MDT0000: new disk, initializing [ 3468.156630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3468.175741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3472.493246] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3482.870964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.050422] Lustre: 112827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3483.104389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3483.108707] Lustre: Skipped 1 previous similar message [ 3483.175561] Lustre: lustre-MDT0001: new disk, initializing [ 3483.277789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3483.321790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3483.331137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3487.708179] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3497.151200] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3502.667077] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3508.608943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3508.869554] Lustre: lustre-OST0000: new disk, initializing [ 3508.872898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3508.878624] Lustre: 114796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3508.949357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3514.586086] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3515.431800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3515.446069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3515.548872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3523.145473] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3527.387948] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3527.622809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3532.246486] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3532.478892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3536.357727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3536.357777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.377203] LustreError: Skipped 1 previous similar message [ 3536.397548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3536.431935] Lustre: Skipped 1 previous similar message [ 3540.359546] Lustre: server umount lustre-OST0000 complete [ 3545.057237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3545.078704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.094730] Lustre: Skipped 1 previous similar message [ 3545.099718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3546.607613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.609481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3546.638389] Lustre: Skipped 1 previous similar message [ 3551.712299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3551.721377] Lustre: Skipped 1 previous similar message [ 3556.832673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3556.839091] Lustre: Skipped 1 previous similar message [ 3558.367248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3558.525322] Lustre: server umount lustre-MDT0000 complete [ 3561.859131] LustreError: 112820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789118933 with bad export cookie 9183368114478241206 [ 3561.873414] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3561.874138] LustreError: 112820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3562.195676] Lustre: server umount lustre-MDT0001 complete [ 3570.862520] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:29:00 (1789118940) [ 3575.840745] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3594.077983] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3604.529681] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3642.070275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3642.277672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3642.301604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3642.366856] Lustre: lustre-MDT0000: new disk, initializing [ 3642.436078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3642.447924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3646.205946] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3655.470087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.570277] Lustre: 119705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3655.624467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3655.634550] Lustre: Skipped 1 previous similar message [ 3655.702118] Lustre: lustre-MDT0001: new disk, initializing [ 3655.776653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3655.813922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3655.834953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3659.150085] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3667.965871] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3673.138293] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3680.009342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3680.275575] Lustre: lustre-OST0000: new disk, initializing [ 3680.279893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3680.287705] Lustre: 121673:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3680.381584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3686.625693] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3690.591712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3690.622705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3690.715750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3696.589866] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3702.182064] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3702.341928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3706.479852] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3706.658583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3711.468885] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3711.470940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.484520] LustreError: Skipped 1 previous similar message [ 3711.485354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3711.508335] Lustre: Skipped 1 previous similar message [ 3714.697952] Lustre: server umount lustre-OST0000 complete [ 3721.697345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.710802] Lustre: Skipped 1 previous similar message [ 3724.055202] Lustre: server umount lustre-MDT0000 complete [ 3727.395341] LustreError: 120510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119099 with bad export cookie 9183368114478242088 [ 3727.401102] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3727.413103] LustreError: 120510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3727.715887] Lustre: server umount lustre-MDT0001 complete [ 3734.984922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.344318] LustreError: 123891:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3735.364848] LustreError: 123891:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3735.404144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.803836] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3740.649496] LustreError: 123892:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3745.765049] LustreError: 123891:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3747.449502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3752.370291] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3760.018329] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3765.391604] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3772.263619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3772.401761] LustreError: 125711:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.089771] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3786.334192] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3793.438526] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3798.496296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3798.500403] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3798.508488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3798.511881] Lustre: Skipped 3 previous similar messages [ 3804.198925] Lustre: server umount lustre-OST0000 complete [ 3809.249647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3814.052948] Lustre: server umount lustre-MDT0000 complete [ 3816.931711] LustreError: 125201:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3816.958734] LustreError: 125201:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3817.545166] LustreError: 123873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119189 with bad export cookie 9183368114478242970 [ 3817.545843] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3817.581734] LustreError: 123873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3817.930885] Lustre: server umount lustre-MDT0001 complete [ 3830.803503] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3833.549426] Key type lgssc unregistered [ 3833.876860] LNet: 127524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3833.884933] LNetError: 127524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3833.894576] LNet: Removed LNI 192.168.203.104@tcp [ 3834.811900] Key type .llcrypt unregistered [ 3834.815249] Key type ._llcrypt unregistered [ 3854.695485] Key type ._llcrypt registered [ 3854.704247] Key type .llcrypt registered [ 3854.838437] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3869.627566] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3870.644638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3870.817407] alg: No test for adler32 (adler32-zlib) [ 3872.060475] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 3872.318034] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3874.023416] Key type lgssc registered [ 3875.364333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3909.266135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3909.298927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.672211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3910.746729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3910.870345] Lustre: lustre-MDT0000: new disk, initializing [ 3910.927120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3910.942691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3916.299687] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3928.896161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.036663] Lustre: 130679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3929.082202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3929.086409] Lustre: Skipped 1 previous similar message [ 3929.169964] Lustre: lustre-MDT0001: new disk, initializing [ 3929.274132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3929.330606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3929.367454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3935.119523] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3947.228652] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3955.614110] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3963.597847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.771533] Lustre: lustre-OST0000: new disk, initializing [ 3963.773890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3963.777547] Lustre: 132649:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3963.856878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.073141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3969.090749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3969.201996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3970.168929] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3979.549757] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3983.939450] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3984.134724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3988.395940] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3988.614711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3995.111162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3995.113787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.114826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3995.128507] LustreError: Skipped 1 previous similar message [ 3995.173144] Lustre: Skipped 1 previous similar message [ 3996.918894] Lustre: server umount lustre-OST0000 complete [ 4005.347316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.355452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.395371] Lustre: Skipped 1 previous similar message [ 4005.416389] Lustre: Skipped 2 previous similar messages [ 4009.176791] Lustre: server umount lustre-MDT0000 complete [ 4014.111042] LustreError: 132648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119385 with bad export cookie 570203269016922131 [ 4014.124414] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4014.138628] LustreError: 132648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4014.402023] Lustre: server umount lustre-MDT0001 complete [ 4023.139405] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:36:33 (1789119393) [ 4029.790045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.198693] LustreError: 135060:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.213522] LustreError: 135060:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4030.277367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4034.776649] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4035.552710] LustreError: 135061:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.652184] LustreError: 135060:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.072574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.341944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4047.469596] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4054.608499] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4060.569947] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4062.765303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4062.776930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4062.783413] Lustre: Skipped 1 previous similar message [ 4067.809325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4067.821509] Lustre: Skipped 1 previous similar message [ 4069.127581] Lustre: server umount lustre-MDT0000 complete [ 4072.914294] LustreError: 135041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119444 with bad export cookie 570203269016923013 [ 4072.920244] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4072.925028] LustreError: 135041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4073.269317] Lustre: server umount lustre-MDT0001 complete [ 4080.683322] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:37:31 (1789119451) [ 4087.136176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4102.688602] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4107.810231] LustreError: 137477:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 4133.535485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4138.951850] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4147.825818] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4149.770413] Lustre: server umount lustre-OST0000 complete [ 4158.572653] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:38:48 (1789119528) [ 4165.824079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4166.342871] LustreError: 138921:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.445152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4171.361641] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4171.747232] LustreError: 138922:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.987850] LustreError: 138922:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.021976] LustreError: 138922:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4182.060872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.838298] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4195.246386] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4200.232562] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4205.714115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4206.000461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.004043] Lustre: Skipped 1 previous similar message [ 4212.336507] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4220.442796] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4228.584819] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.593102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4228.596684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4228.596700] Lustre: Skipped 2 previous similar messages [ 4228.604541] Lustre: Skipped 3 previous similar messages [ 4234.333963] Lustre: server umount lustre-MDT0000 complete [ 4237.908429] LustreError: 138901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119609 with bad export cookie 570203269016923678 [ 4237.915329] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4237.924708] LustreError: 138901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4238.171689] Lustre: server umount lustre-MDT0001 complete [ 4252.117477] Lustre: server umount lustre-OST0000 complete [ 4261.284532] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:40:31 (1789119631) [ 4268.564052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.005095] LustreError: 142575:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4273.737584] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4281.878448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4282.227116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4282.247139] Lustre: Skipped 1 previous similar message [ 4286.999326] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4293.839294] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4299.296859] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4306.128166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4313.422114] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4313.779538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4321.249272] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4325.831832] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4326.016629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4330.523566] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4330.688717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4333.536295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4333.542662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4333.554146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4333.562244] Lustre: Skipped 5 previous similar messages [ 4338.648890] Lustre: server umount lustre-OST0000 complete [ 4343.783784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4343.793634] LustreError: Skipped 1 previous similar message [ 4343.799941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4343.814778] Lustre: Skipped 1 previous similar message [ 4348.290962] Lustre: server umount lustre-MDT0000 complete [ 4349.922163] LustreError: 142576:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.938804] LustreError: 142576:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4351.669061] LustreError: 142557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119723 with bad export cookie 570203269016924749 [ 4351.672783] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4351.678735] LustreError: 142557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4351.997474] Lustre: server umount lustre-MDT0001 complete [ 4359.835963] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:42:10 (1789119730) [ 4366.604287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4382.175787] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4387.295609] LustreError: 146733:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 4413.023777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4413.030379] Lustre: Skipped 1 previous similar message [ 4419.797193] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4427.940269] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4435.808786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.448594] LustreError: 146758:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.575646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4441.767303] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4450.869383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.233448] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4464.710996] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4471.049330] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4475.986145] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4476.249436] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4481.474705] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4481.797880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4487.137152] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4487.145316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4487.157449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4487.157456] Lustre: Skipped 5 previous similar messages [ 4487.182416] Lustre: Skipped 2 previous similar messages [ 4490.885917] Lustre: server umount lustre-MDT0000 complete [ 4495.315918] LustreError: 146739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789119867 with bad export cookie 570203269016925771 [ 4495.319283] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4495.338728] LustreError: 146739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4495.685473] Lustre: server umount lustre-MDT0001 complete [ 4510.827624] Lustre: server umount lustre-OST0000 complete [ 4520.568480] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:44:50 (1789119890) [ 4527.432723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4543.013157] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4548.127450] LustreError: 150930:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 4580.890981] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4589.744530] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4597.183784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.747944] LustreError: 150955:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4597.780722] LustreError: 150955:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 4597.918892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4603.179872] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4611.610351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.672403] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4626.098442] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4632.616465] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4640.437293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4640.629571] Lustre: lustre-OST0001: new disk, initializing [ 4640.640370] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4640.649891] Lustre: 153803:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4647.987871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4648.011222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4648.200514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4648.510160] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4658.789563] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4664.079076] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4664.331000] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4668.675509] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4668.922045] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4673.505678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4673.523815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.547247] Lustre: Skipped 1 previous similar message [ 4673.555596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4673.561763] Lustre: Skipped 2 previous similar messages [ 4676.829230] Lustre: server umount lustre-OST0000 complete [ 4695.007126] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4695.253394] Lustre: server umount lustre-OST0001 complete [ 4704.225739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4704.232850] LustreError: Skipped 2 previous similar messages [ 4705.556698] Lustre: server umount lustre-MDT0000 complete [ 4708.957514] LustreError: 150937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120080 with bad export cookie 570203269016926723 [ 4708.965081] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.971929] LustreError: 150937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4709.467239] Lustre: server umount lustre-MDT0001 complete [ 4731.135141] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:48:21 (1789120101) [ 4732.839548] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4734.475899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4735.948992] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:48:26 (1789120106) [ 4742.406956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.663937] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4742.697545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4742.898042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4742.901216] Lustre: Skipped 6 previous similar messages [ 4747.167505] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4755.506280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.723327] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4755.798121] Lustre: 157366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4760.625896] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4769.661472] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4775.847395] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4784.394739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4784.733533] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4791.774923] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4792.466934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4800.161792] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4805.164235] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4805.449423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4809.413558] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4809.594581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4812.769916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4812.783978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.814013] Lustre: Skipped 5 previous similar messages [ 4812.827588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4812.836697] Lustre: Skipped 11 previous similar messages [ 4818.372884] Lustre: server umount lustre-OST0000 complete [ 4844.063175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4852.676143] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4853.750618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4853.771023] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4861.815832] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4864.040701] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4864.071492] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4866.802375] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4867.073271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4873.049900] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4873.289454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4879.774921] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99ef987f4800.ost_server_uuid 50 [ 4881.199160] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff99ef987f4800.ost_server_uuid in FULL state after 0 sec [ 4881.414944] LustreError: 157376:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.4@tcp) returned error from blocking AST (req@0000000010947ba2 x1876027500666496 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000aa95de3b/0x7e9c4da2385435d lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.4@tcp remote: 0xa43103b0c6f08187 expref: 5 pid: 157374 timeout: 4981 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4881.444959] LustreError: lustre-MDT0000: A client on nid 192.168.203.4@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4881.452257] LustreError: 157364:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.4@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000aa95de3b/0x7e9c4da2385435d lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.4@tcp remote: 0xa43103b0c6f08187 expref: 6 pid: 157374 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4887.012657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4887.026180] LustreError: Skipped 1 previous similar message [ 4892.267385] Lustre: server umount lustre-OST0000 complete [ 4902.050693] Lustre: server umount lustre-MDT0000 complete [ 4904.933048] LustreError: 157380:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.963894] LustreError: 157380:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 4905.387641] LustreError: 159242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120277 with bad export cookie 570203269016927906 [ 4905.399738] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4905.400205] LustreError: 159242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4911.862965] Lustre: server umount lustre-MDT0001 complete [ 4921.677897] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4924.070863] Key type lgssc unregistered [ 4924.288545] LNet: 163485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4924.298895] LNetError: 163485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4924.311618] LNet: Removed LNI 192.168.203.104@tcp [ 4925.324286] Key type .llcrypt unregistered [ 4925.330211] Key type ._llcrypt unregistered [ 4950.501378] Key type ._llcrypt registered [ 4950.504475] Key type .llcrypt registered [ 4950.717731] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4952.250719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4952.564794] alg: No test for adler32 (adler32-zlib) [ 4953.753445] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 4953.961484] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4955.649845] Key type lgssc registered [ 4956.930338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4964.909510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4964.921792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.489286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.018202] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4976.609286] LustreError: 164902:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4976.629893] LustreError: 164902:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4979.777191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4980.078803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4983.852517] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4990.343090] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4995.289220] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5000.708200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5001.019105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5006.239652] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5011.265444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5013.217776] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5019.194031] Lustre: Failing over lustre-MDT0000 [ 5019.452555] Lustre: server umount lustre-MDT0000 complete [ 5021.152109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5021.159849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.184703] LustreError: 164902:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5021.665157] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.681135] Lustre: Skipped 1 previous similar message [ 5026.785733] LustreError: 164901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.806820] LustreError: 164901:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 5031.297521] LustreError: 164881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.4@tcp arrived at 1789120403 with bad export cookie 9799133386987308704 [ 5031.314698] LustreError: 164881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5031.914578] LustreError: 164901:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5031.937974] LustreError: 164901:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5037.031939] LustreError: 164902:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5037.059874] LustreError: 164902:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5038.048743] Lustre: 164211:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789120393/real 1789120393] req@00000000499596a2 x1876028633932288/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789120409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5038.075349] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5038.326496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.264777] LustreError: 164210:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000020fc001c x1876028633937536/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5047.296252] LustreError: 164902:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5047.329671] LustreError: 164902:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 5047.535930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5047.597217] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5051.938795] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5052.906779] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5052.925324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5054.272815] Lustre: lustre-MDT0000: Denying connection for new client 9f3b727b-ee2c-407a-953f-9d76887b3da4 (at 192.168.203.4@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5063.145716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5063.152512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5063.154844] Lustre: Skipped 1 previous similar message [ 5063.174276] Lustre: Skipped 1 previous similar message [ 5068.260565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5068.269914] Lustre: Skipped 1 previous similar message [ 5068.633846] Lustre: server umount lustre-OST0000 complete [ 5072.567182] LustreError: 168634:0:(ldlm_lib.c:3067:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5072.588612] Lustre: 167981:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5072.596506] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5072.624070] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5072.633154] LustreError: 167981:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001eeb1f8c x1876028633951872/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5072.652385] LustreError: 167981:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5072.667577] LustreError: 167981:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5072.680682] Lustre: lustre-MDT0000: Recovery over after 0:20, of 2 clients 0 recovered and 2 were evicted. [ 5073.020704] Lustre: server umount lustre-MDT0000 complete [ 5073.379531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5073.394060] LustreError: 164897:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5076.673588] LustreError: 164882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120448 with bad export cookie 9799133386987308802 [ 5076.682856] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5076.685809] LustreError: 164882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5076.703734] LustreError: 164210:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5079.179824] Lustre: server umount lustre-MDT0001 complete [ 5092.580339] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5095.408386] Key type lgssc unregistered [ 5095.788691] LNet: 169390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5095.795865] LNetError: 169390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5096.885356] LNet: Removed LNI 192.168.203.104@tcp [ 5098.071768] Key type .llcrypt unregistered [ 5098.078912] Key type ._llcrypt unregistered [ 5125.614297] Key type ._llcrypt registered [ 5125.617487] Key type .llcrypt registered [ 5125.747369] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5126.943345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5127.051151] alg: No test for adler32 (adler32-zlib) [ 5128.226125] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 5128.488608] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5130.183194] Key type lgssc registered [ 5131.263625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5139.906599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5139.943868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.437264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5146.665108] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5151.713551] LustreError: 170804:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.739968] LustreError: 170804:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5155.778885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5156.358921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5161.619316] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5169.821753] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5176.271701] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5183.422156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5183.812385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5186.872912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5190.812425] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5198.604499] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5207.528299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5207.538786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5207.559500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5207.564402] Lustre: Skipped 1 previous similar message [ 5212.293081] Lustre: server umount lustre-OST0000 complete [ 5217.773379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.781782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.789130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5217.789138] LustreError: Skipped 1 previous similar message [ 5217.796230] Lustre: Skipped 2 previous similar messages [ 5217.821248] Lustre: Skipped 1 previous similar message [ 5222.464832] Lustre: server umount lustre-MDT0000 complete [ 5222.881810] LustreError: 170804:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5222.895182] LustreError: 170804:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5226.554279] LustreError: 170787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120598 with bad export cookie 7287855728098966268 [ 5226.564295] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5226.576131] LustreError: 170787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5226.993597] Lustre: server umount lustre-MDT0001 complete [ 5238.187183] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5241.025977] Key type lgssc unregistered [ 5241.350299] LNet: 174342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5241.360654] LNetError: 174342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5241.398657] LNet: Removed LNI 192.168.203.104@tcp [ 5242.338629] Key type .llcrypt unregistered [ 5242.341780] Key type ._llcrypt unregistered [ 5278.888554] Key type ._llcrypt registered [ 5278.892409] Key type .llcrypt registered [ 5279.042029] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5280.184599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5280.264026] alg: No test for adler32 (adler32-zlib) [ 5281.469256] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 5281.756930] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5283.473057] Key type lgssc registered [ 5284.980239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5294.732275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5294.774170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5296.348262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.739544] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5306.848923] LustreError: 176053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.875895] LustreError: 176053:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5311.828217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.971419] LustreError: 176052:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5312.303221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5317.427962] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5327.408700] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5333.901088] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5341.353936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5341.754166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5342.782872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5349.170420] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5357.569776] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5368.289704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5368.299653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.304464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5368.320087] Lustre: Skipped 1 previous similar message [ 5368.351064] Lustre: Skipped 1 previous similar message [ 5370.529711] Lustre: server umount lustre-OST0000 complete [ 5378.529868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.532955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.541851] Lustre: Skipped 1 previous similar message [ 5378.558373] Lustre: Skipped 1 previous similar message [ 5380.732618] Lustre: server umount lustre-MDT0000 complete [ 5383.650467] LustreError: 176780:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5383.673808] LustreError: 176780:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5385.389700] LustreError: 176034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120757 with bad export cookie 11084465299080703320 [ 5385.401482] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5385.409644] LustreError: 176034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5385.865236] Lustre: server umount lustre-MDT0001 complete [ 5399.615748] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5405.246559] Key type lgssc unregistered [ 5405.582344] LNet: 179588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5405.602424] LNetError: 179588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5405.634353] LNet: Removed LNI 192.168.203.104@tcp [ 5407.091204] Key type .llcrypt unregistered [ 5407.102308] Key type ._llcrypt unregistered [ 5431.908073] Key type ._llcrypt registered [ 5431.911487] Key type .llcrypt registered [ 5432.031545] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5433.046487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5433.082618] alg: No test for adler32 (adler32-zlib) [ 5434.099288] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 5434.349446] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5436.056046] Key type lgssc registered [ 5437.045431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5449.615903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5449.642497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.010899] Lustre: *** cfs_fail_loc=135, val=0*** [ 5451.104782] LustreError: 181114:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5451.114862] LustreError: 181114:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -2 [ 5451.127029] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5451.127029] [ 5451.133187] LustreError: MGC192.168.203.104@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5451.155322] LustreError: 181075:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5451.170225] LustreError: 181075:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5451.189668] LustreError: 181075:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5451.195448] LustreError: 181075:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5451.333746] Lustre: server umount lustre-MDT0000 complete [ 5451.339347] LustreError: 181075:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5458.873719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5459.757453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.653886] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5470.235975] LustreError: 181486:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5470.262890] LustreError: 181486:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5475.298297] LustreError: 181485:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.127982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5476.555984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5481.140806] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5489.793435] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5496.977980] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5501.281963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5501.297625] Lustre: Skipped 1 previous similar message [ 5501.298713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5501.314781] Lustre: Skipped 1 previous similar message [ 5505.590799] Lustre: server umount lustre-MDT0000 complete [ 5506.531451] LustreError: 182214:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.548680] LustreError: 182214:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5509.644281] LustreError: 181466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789120881 with bad export cookie 5241890788471964851 [ 5509.652435] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5509.662373] LustreError: 181466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5510.170917] Lustre: server umount lustre-MDT0001 complete [ 5522.021675] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5524.661675] Key type lgssc unregistered [ 5524.955381] LNet: 183822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5524.965157] LNetError: 183822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5524.993730] LNet: Removed LNI 192.168.203.104@tcp [ 5526.104630] Key type .llcrypt unregistered [ 5526.107808] Key type ._llcrypt unregistered [ 5549.956492] Key type ._llcrypt registered [ 5549.959331] Key type .llcrypt registered [ 5550.091216] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5550.563362] Key type .llcrypt unregistered [ 5550.565884] Key type ._llcrypt unregistered [ 5567.287431] Key type ._llcrypt registered [ 5567.290794] Key type .llcrypt registered [ 5567.364159] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5568.885932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5568.921409] alg: No test for adler32 (adler32-zlib) [ 5569.917189] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 5570.187582] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5571.935186] Key type lgssc registered [ 5573.268866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5581.951258] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5581.968723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5597.535392] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5602.658127] LustreError: 185959:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 5629.343202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.863971] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5645.614520] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5652.746747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5653.272151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5653.334766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5658.468146] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5658.605340] LustreError: 185984:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5658.634235] LustreError: 185984:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5663.722640] LustreError: 186999:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5663.735029] LustreError: 186999:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5666.910280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5667.353202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5672.974712] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5681.360383] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5687.531740] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5690.316358] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5703.140425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5703.158114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.182220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5708.261665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5708.270046] Lustre: Skipped 2 previous similar messages [ 5708.541789] Lustre: server umount lustre-OST0000 complete [ 5713.381097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.382918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5713.392981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5713.397486] Lustre: Skipped 2 previous similar messages [ 5718.501265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5718.512751] Lustre: Skipped 2 previous similar messages [ 5718.803428] Lustre: server umount lustre-MDT0000 complete [ 5722.720223] LustreError: 185966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121094 with bad export cookie 14983884613985502709 [ 5722.726764] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5722.735323] LustreError: 185966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5723.076889] Lustre: server umount lustre-MDT0001 complete [ 5734.860881] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5737.142270] Key type lgssc unregistered [ 5737.436863] LNet: 189976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5737.445252] LNetError: 189976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5737.463784] LNet: Removed LNI 192.168.203.104@tcp [ 5738.672347] Key type .llcrypt unregistered [ 5738.680197] Key type ._llcrypt unregistered [ 5766.006402] Key type ._llcrypt registered [ 5766.010086] Key type .llcrypt registered [ 5766.107660] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5766.764683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5766.802243] alg: No test for adler32 (adler32-zlib) [ 5767.961245] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 5768.177224] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5769.935768] Key type lgssc registered [ 5771.030118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5778.827640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5778.857161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.276946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.284600] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5790.704214] LustreError: 191392:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.719113] LustreError: 191392:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5794.936836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.537690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5800.903233] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5808.863971] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5814.887490] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5822.453368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5822.841209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.942847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5829.049880] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5837.233512] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5842.545126] Lustre: Failing over lustre-MDT0000 [ 5842.903778] Lustre: server umount lustre-MDT0000 complete [ 5844.481187] LustreError: 191388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.478789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5845.509118] Lustre: Skipped 1 previous similar message [ 5845.522199] LustreError: 191391:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.552494] LustreError: 191391:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5849.571259] LustreError: 193401:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5854.698587] LustreError: 191388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5854.732459] LustreError: 191388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5861.855142] Lustre: 190706:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789121217/real 1789121217] req@00000000b5225d6a x1876029487473792/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789121233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5861.906198] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5863.266247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5864.930516] LustreError: 191388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.959420] LustreError: 191388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 5872.097988] LustreError: 190702:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000499596a2 x1876029487478784/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5872.397652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5872.505759] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5876.194043] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5877.008722] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5877.749458] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5877.794306] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5877.870332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5881.946594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5888.353723] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5896.675808] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5896.691069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5896.705930] Lustre: Skipped 1 previous similar message [ 5896.718577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5898.218996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5898.220851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5902.662986] Lustre: server umount lustre-OST0000 complete [ 5908.449209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5908.457717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5908.470100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5908.470107] Lustre: Skipped 1 previous similar message [ 5912.923939] Lustre: server umount lustre-MDT0000 complete [ 5913.577083] LustreError: 192120:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5913.612085] LustreError: 192120:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 5917.212178] LustreError: 191372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121288 with bad export cookie 9724917219801860478 [ 5917.228493] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5917.236069] LustreError: 191372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5917.713050] Lustre: server umount lustre-MDT0001 complete [ 5930.315145] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5933.350217] Key type lgssc unregistered [ 5933.719303] LNet: 196236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5933.728417] LNetError: 196236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5933.764637] LNet: Removed LNI 192.168.203.104@tcp [ 5934.716605] Key type .llcrypt unregistered [ 5934.718907] Key type ._llcrypt unregistered [ 5959.085830] Key type ._llcrypt registered [ 5959.088375] Key type .llcrypt registered [ 5959.182056] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5960.250953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5960.411508] alg: No test for adler32 (adler32-zlib) [ 5961.546124] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 5961.805062] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5963.495172] Key type lgssc registered [ 5964.678944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5972.651879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5972.666333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.947579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.330674] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5984.228096] LustreError: 197653:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5984.242942] LustreError: 197653:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5987.749294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5988.257491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5993.491417] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6001.610318] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6007.071145] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6014.162827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6014.681691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.831420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6021.983377] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6030.578771] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6034.858194] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6041.720476] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6056.616470] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6069.234434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.248613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6069.252280] Lustre: Skipped 1 previous similar message [ 6069.253484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6069.266246] Lustre: Skipped 1 previous similar message [ 6074.000696] Lustre: server umount lustre-OST0000 complete [ 6079.463092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.483908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6079.495508] Lustre: Skipped 1 previous similar message [ 6084.466843] Lustre: server umount lustre-MDT0000 complete [ 6084.577780] LustreError: 198381:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6084.593770] LustreError: 198381:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6088.237281] LustreError: 197635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121460 with bad export cookie 6614759192160169757 [ 6088.244878] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6088.267684] LustreError: 197635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6088.590869] Lustre: server umount lustre-MDT0001 complete [ 6100.949690] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6104.065092] Key type lgssc unregistered [ 6104.363053] LNet: 201504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6104.377421] LNetError: 201504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6104.405563] LNet: Removed LNI 192.168.203.104@tcp [ 6105.441876] Key type .llcrypt unregistered [ 6105.443716] Key type ._llcrypt unregistered [ 6126.724282] Key type ._llcrypt registered [ 6126.734723] Key type .llcrypt registered [ 6126.910888] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6128.494826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6128.661382] alg: No test for adler32 (adler32-zlib) [ 6129.944788] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 6130.363527] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6132.207178] Key type lgssc registered [ 6133.335481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6142.993973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6143.011258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.694180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6149.311417] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6155.237262] LustreError: 202773:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.268451] LustreError: 202773:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6158.395695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.856114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6163.640469] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6171.774839] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6177.434637] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6184.349696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6184.738540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6190.240833] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6190.889296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6197.174796] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6203.592420] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6216.162455] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6225.516289] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6234.892965] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6245.347222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6245.375795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6245.385099] Lustre: Skipped 1 previous similar message [ 6250.471625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6250.483486] Lustre: Skipped 1 previous similar message [ 6250.740488] Lustre: server umount lustre-OST0000 complete [ 6255.589057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.609259] Lustre: Skipped 1 previous similar message [ 6255.615505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6255.620601] Lustre: Skipped 1 previous similar message [ 6260.705445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6260.721700] Lustre: Skipped 1 previous similar message [ 6265.825404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6265.840401] Lustre: Skipped 1 previous similar message [ 6268.383148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6268.529781] Lustre: server umount lustre-MDT0000 complete [ 6270.944913] LustreError: 202768:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.957956] LustreError: 202768:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6272.226816] LustreError: 203508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121644 with bad export cookie 342276855661348524 [ 6272.227949] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6272.238054] LustreError: 203508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6272.659153] Lustre: server umount lustre-MDT0001 complete [ 6283.500864] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6285.966997] Key type lgssc unregistered [ 6286.241559] LNet: 207893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6286.246914] LNetError: 207893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6286.273345] LNet: Removed LNI 192.168.203.104@tcp [ 6287.043912] Key type .llcrypt unregistered [ 6287.047421] Key type ._llcrypt unregistered [ 6310.175983] Key type ._llcrypt registered [ 6310.180603] Key type .llcrypt registered [ 6310.296290] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6311.312128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6311.341581] alg: No test for adler32 (adler32-zlib) [ 6312.448615] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 6312.685756] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6314.375178] Key type lgssc registered [ 6315.463859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6322.227613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6322.253893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6323.654419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6328.688805] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6333.920759] LustreError: 209309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6333.933931] LustreError: 209309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6335.780682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.087723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6340.592747] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6347.541764] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6352.907840] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6359.240265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6359.632490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.223833] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6368.944323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6372.268268] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6386.656531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6386.664876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.676205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6389.728641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6389.743151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6389.754222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.675980] Lustre: server umount lustre-OST0000 complete [ 6397.408459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6397.421859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6397.446356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6397.456742] Lustre: Skipped 1 previous similar message [ 6399.972516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6399.976514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.912232] Lustre: server umount lustre-MDT0000 complete [ 6405.101962] LustreError: 209308:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.143626] LustreError: 209308:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6405.855825] LustreError: 211802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121777 with bad export cookie 3841858133203758535 [ 6405.865736] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6405.868224] LustreError: 211802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6406.154541] Lustre: server umount lustre-MDT0001 complete [ 6417.588820] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6419.889860] Key type lgssc unregistered [ 6420.180143] LNet: 212995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6420.187935] LNetError: 212995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6420.202738] LNet: Removed LNI 192.168.203.104@tcp [ 6421.077168] Key type .llcrypt unregistered [ 6421.093329] Key type ._llcrypt unregistered [ 6444.542451] Key type ._llcrypt registered [ 6444.545153] Key type .llcrypt registered [ 6444.654213] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6445.789617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6445.909874] alg: No test for adler32 (adler32-zlib) [ 6447.109895] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 6447.384066] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6449.095201] Key type lgssc registered [ 6450.304194] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6457.729370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6457.774776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6459.471676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6463.823395] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6469.600311] LustreError: 214264:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6469.620155] LustreError: 214264:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6473.235145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6473.535874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6477.822297] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6486.103225] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6492.186775] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6499.646288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6499.925583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.073690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6505.555246] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6512.106961] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6524.900538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6524.918669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6530.021583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6530.031594] Lustre: Skipped 1 previous similar message [ 6530.760663] Lustre: server umount lustre-OST0000 complete [ 6535.141156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6535.143615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.157138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6535.168885] Lustre: Skipped 2 previous similar messages [ 6535.180077] Lustre: Skipped 1 previous similar message [ 6540.258450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6540.263810] Lustre: Skipped 2 previous similar messages [ 6540.469964] Lustre: server umount lustre-MDT0000 complete [ 6543.629127] LustreError: 214245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789121915 with bad export cookie 1016448990476509870 [ 6543.635741] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6543.643319] LustreError: 214245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6543.966689] Lustre: server umount lustre-MDT0001 complete [ 6554.312808] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6557.006340] Key type lgssc unregistered [ 6557.310468] LNet: 217949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6557.316683] LNetError: 217949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6557.328842] LNet: Removed LNI 192.168.203.104@tcp [ 6558.161884] Key type .llcrypt unregistered [ 6558.164338] Key type ._llcrypt unregistered [ 6577.592649] Key type ._llcrypt registered [ 6577.597561] Key type .llcrypt registered [ 6577.716412] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6578.756089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6578.783859] alg: No test for adler32 (adler32-zlib) [ 6579.762710] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 6579.987582] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6581.631299] Key type lgssc registered [ 6582.739175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6589.733939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6589.765967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6591.229747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6596.476465] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6600.680833] LustreError: 219216:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6600.725221] LustreError: 219216:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6605.805876] LustreError: 219215:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6609.543459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6610.388437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6616.479769] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6625.511347] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6631.442965] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6638.443923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6638.977137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6644.639945] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6651.370758] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6660.467551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6670.556841] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6670.646070] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6670.933880] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6677.041215] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6684.192564] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6685.570889] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6697.537323] Lustre: Permanently deactivating lustre-OST0001 [ 6697.550786] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6705.194716] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6713.729093] Lustre: Permanently reactivating lustre-OST0001 [ 6713.752960] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6713.757346] Lustre: Skipped 2 previous similar messages [ 6714.489105] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6714.505585] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6714.518623] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6714.533211] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6714.595588] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6725.109589] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6725.115060] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6725.125036] Lustre: Skipped 1 previous similar message [ 6725.128863] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6728.176037] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6728.182838] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6731.285928] Lustre: server umount lustre-OST0001 complete [ 6733.307214] LustreError: 221040:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6733.337184] LustreError: 221040:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6736.356520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6736.367380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.383921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6736.391800] Lustre: Skipped 1 previous similar message [ 6737.378895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6738.411317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6738.411721] LustreError: 221710:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6738.416261] Lustre: Skipped 2 previous similar messages [ 6738.431384] LustreError: 221710:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6741.675578] Lustre: server umount lustre-OST0000 complete [ 6748.644518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6748.664322] Lustre: Skipped 1 previous similar message [ 6748.673774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6748.682582] Lustre: Skipped 1 previous similar message [ 6751.755768] Lustre: server umount lustre-MDT0000 complete [ 6753.760234] LustreError: 219216:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6753.779697] LustreError: 219216:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6755.527947] LustreError: 219195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122127 with bad export cookie 16618163865653919431 [ 6755.531634] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6755.546182] LustreError: 219195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6756.015713] Lustre: server umount lustre-MDT0001 complete [ 6769.564828] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6773.629577] Key type lgssc unregistered [ 6774.045740] LNet: 224310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6774.059327] LNetError: 224310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6775.077118] LNet: Removed LNI 192.168.203.104@tcp [ 6776.394145] Key type .llcrypt unregistered [ 6776.395889] Key type ._llcrypt unregistered [ 6815.791561] Key type ._llcrypt registered [ 6815.794607] Key type .llcrypt registered [ 6815.951442] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6817.178732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6817.211606] alg: No test for adler32 (adler32-zlib) [ 6818.289109] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 6818.517424] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6820.191519] Key type lgssc registered [ 6821.425948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6828.659936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6828.677332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.969741] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6830.002898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6830.142191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6835.181192] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6844.044935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.198725] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6844.236954] Lustre: 226519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6844.462713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6849.426651] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6857.305616] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6862.807897] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6869.076359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6869.321308] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6869.570134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6875.893194] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6878.396214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6883.542585] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6887.749897] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6893.315874] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6897.802986] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6906.139590] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6921.025505] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6921.037502] Lustre: Skipped 1 previous similar message [ 6940.167838] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6940.173142] Lustre: Skipped 2 previous similar messages [ 6974.615247] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6974.637334] Lustre: Skipped 4 previous similar messages [ 7047.655181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7047.659418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7047.687666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7047.692256] Lustre: Skipped 1 previous similar message [ 7052.644165] Lustre: server umount lustre-OST0000 complete [ 7057.889459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.899364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7057.899768] Lustre: Skipped 2 previous similar messages [ 7057.918333] Lustre: Skipped 1 previous similar message [ 7062.473590] Lustre: server umount lustre-MDT0000 complete [ 7066.024386] LustreError: 226510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122437 with bad export cookie 5606743280442373977 [ 7066.026589] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7066.038021] LustreError: 226510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7066.392864] Lustre: server umount lustre-MDT0001 complete [ 7077.298529] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7079.860768] Key type lgssc unregistered [ 7080.124731] LNet: 231310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7080.133157] LNetError: 231310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7080.157784] LNet: Removed LNI 192.168.203.104@tcp [ 7080.933346] Key type .llcrypt unregistered [ 7080.936404] Key type ._llcrypt unregistered [ 7105.350551] Key type ._llcrypt registered [ 7105.353081] Key type .llcrypt registered [ 7105.437306] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7106.084875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7106.097101] alg: No test for adler32 (adler32-zlib) [ 7107.155343] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 7107.380733] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7109.048049] Key type lgssc registered [ 7110.281678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7118.389632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7118.413731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7120.154522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7125.551654] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7129.571889] LustreError: 232738:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7129.585461] LustreError: 232738:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7134.505808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.689770] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7134.857759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7139.366626] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7146.870772] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7153.739887] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7161.810911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7162.314684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7164.404101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7169.754402] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7176.744687] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7195.109994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.116537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7195.129955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7195.142693] Lustre: Skipped 1 previous similar message [ 7195.537458] Lustre: server umount lustre-OST0000 complete [ 7201.248868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7201.253320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7201.277147] Lustre: Skipped 1 previous similar message [ 7201.279073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7206.288764] Lustre: server umount lustre-MDT0000 complete [ 7206.368264] LustreError: 232738:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.379730] LustreError: 232738:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7210.088636] LustreError: 232718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789122581 with bad export cookie 6618552952901832439 [ 7210.096630] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7210.457164] Lustre: server umount lustre-MDT0001 complete [ 7220.609881] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7223.223363] Key type lgssc unregistered [ 7223.515808] LNet: 236439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7223.523253] LNetError: 236439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7223.544090] LNet: Removed LNI 192.168.203.104@tcp [ 7224.463266] Key type .llcrypt unregistered [ 7224.465739] Key type ._llcrypt unregistered [ 7244.991257] Key type ._llcrypt registered [ 7244.994514] Key type .llcrypt registered [ 7245.135318] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7245.935872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7246.018085] alg: No test for adler32 (adler32-zlib) [ 7247.115355] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 7247.345491] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7249.055199] Key type lgssc registered [ 7250.288584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7520.421332] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7523.060156] Key type lgssc unregistered [ 7523.313795] LNet: 238445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7523.321938] LNetError: 238445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7523.338258] LNet: Removed LNI 192.168.203.104@tcp [ 7524.105522] Key type .llcrypt unregistered [ 7524.110794] Key type ._llcrypt unregistered [ 7548.600534] Key type ._llcrypt registered [ 7548.603304] Key type .llcrypt registered [ 7548.705628] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7568.637680] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7569.480827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7569.495445] alg: No test for adler32 (adler32-zlib) [ 7570.606908] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 7570.811784] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7572.423202] Key type lgssc registered [ 7573.423417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7603.876223] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7347 sec ========== 06:36:13 (1789122973) [ 7606.234409] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:36:15 (1789122975) === [ 7610.003842] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:36:19 (1789122979) === [ 7652.009333] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7654.588530] Key type lgssc unregistered [ 7654.898751] LNet: 243687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7654.911134] LNetError: 243687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7654.929220] LNet: Removed LNI 192.168.203.104@tcp [ 7655.749767] Key type .llcrypt unregistered [ 7655.751981] Key type ._llcrypt unregistered