[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 465282189 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007450] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.021427] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026549] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028063] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030088] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031123] *** VALIDATE tmpfs *** [ 0.033115] *** VALIDATE proc *** [ 0.034298] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036286] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040013] Speculative Store Bypass: Vulnerable [ 0.042900] debug: unmapping init [mem 0xffffffff83e59000-0xffffffff83e60fff] [ 0.044976] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045747] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049025] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053284] rcu: Hierarchical SRCU implementation. [ 0.055434] smp: Bringing up secondary CPUs ... [ 0.056498] x86: Booting SMP configuration: [ 0.057019] .... node #0, CPUs: #1 #2 #3 [ 0.061127] smp: Brought up 1 node, 4 CPUs [ 0.062970] smpboot: Max logical packages: 1 [ 0.063023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137614] node 0 deferred pages initialised in 70ms [ 0.141410] devtmpfs: initialized [ 0.143294] x86/mm: Memory block size: 128MB [ 0.146187] gcov: version magic: 0x41383552 [ 0.149078] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151343] pinctrl core: initialized pinctrl subsystem [ 0.153029] [ 0.153648] ************************************************************* [ 0.156013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158011] ** ** [ 0.161014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164014] ** ** [ 0.166012] ** This means that this kernel is built to expose internal ** [ 0.169012] ** IOMMU data structures, which may compromise security on ** [ 0.172014] ** your system. ** [ 0.175016] ** ** [ 0.177009] ** If you see this message and you are not debugging the ** [ 0.179013] ** kernel, report this immediately to your vendor! ** [ 0.182019] ** ** [ 0.184012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187014] ************************************************************* [ 0.190136] NET: Registered protocol family 16 [ 0.192444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199094] cpuidle: using governor menu [ 0.200737] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203460] PCI: Using configuration type 1 for base access [ 0.205126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214167] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218137] cryptd: max_cpu_qlen set to 1000 [ 0.220148] ACPI: Added _OSI(Module Device) [ 0.221012] ACPI: Added _OSI(Processor Device) [ 0.222039] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.223000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.228225] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235661] ACPI: Interpreter enabled [ 0.237065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239016] ACPI: Using IOAPIC for interrupt routing [ 0.241107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245422] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257068] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260078] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272625] acpiphp: Slot [2] registered [ 0.274183] acpiphp: Slot [5] registered [ 0.276120] acpiphp: Slot [6] registered [ 0.278128] acpiphp: Slot [7] registered [ 0.280110] acpiphp: Slot [8] registered [ 0.281117] acpiphp: Slot [9] registered [ 0.282128] acpiphp: Slot [10] registered [ 0.284152] acpiphp: Slot [3] registered [ 0.286149] acpiphp: Slot [4] registered [ 0.288078] acpiphp: Slot [11] registered [ 0.289086] acpiphp: Slot [12] registered [ 0.291167] acpiphp: Slot [13] registered [ 0.292102] acpiphp: Slot [14] registered [ 0.294113] acpiphp: Slot [15] registered [ 0.296157] acpiphp: Slot [16] registered [ 0.297136] acpiphp: Slot [17] registered [ 0.299088] acpiphp: Slot [18] registered [ 0.300080] acpiphp: Slot [19] registered [ 0.302101] acpiphp: Slot [20] registered [ 0.303000] acpiphp: Slot [21] registered [ 0.303000] acpiphp: Slot [22] registered [ 0.306030] acpiphp: Slot [23] registered [ 0.307119] acpiphp: Slot [24] registered [ 0.309116] acpiphp: Slot [25] registered [ 0.311118] acpiphp: Slot [26] registered [ 0.313197] acpiphp: Slot [27] registered [ 0.315142] acpiphp: Slot [28] registered [ 0.316115] acpiphp: Slot [29] registered [ 0.318135] acpiphp: Slot [30] registered [ 0.320124] acpiphp: Slot [31] registered [ 0.322081] PCI host bridge to bus 0000:00 [ 0.324025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329073] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.333022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.335021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344994] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.349264] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358733] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.363860] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.367022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.370021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378423] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380863] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.386731] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.407015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.412015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.417805] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.435015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.454017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.463694] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.476013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.483016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.507016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.514233] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.522015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.528015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.546016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.555685] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.560012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.565017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.581013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.590445] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.598020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.604906] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.620019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.626782] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.633017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.639015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.663014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.673810] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.676310] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.678256] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.679281] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.682781] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.687138] iommu: Default domain type: Passthrough [ 0.688318] SCSI subsystem initialized [ 0.690127] ACPI: bus type USB registered [ 0.691071] usbcore: registered new interface driver usbfs [ 0.692077] usbcore: registered new interface driver hub [ 0.694061] usbcore: registered new device driver usb [ 0.695150] pps_core: LinuxPPS API ver. 1 registered [ 0.697012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.699068] PTP clock support registered [ 0.701071] EDAC MC: Ver: 3.0.0 [ 0.702363] PCI: Using ACPI for IRQ routing [ 0.703996] NetLabel: Initializing [ 0.704012] NetLabel: domain hash size = 128 [ 0.705015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.706098] NetLabel: unlabeled traffic allowed by default [ 0.708049] vgaarb: loaded [ 0.709285] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.711014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.714416] clocksource: Switched to clocksource kvm-clock [ 0.820151] VFS: Disk quotas dquot_6.6.0 [ 0.821569] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.824375] *** VALIDATE ramfs *** [ 0.825788] *** VALIDATE hugetlbfs *** [ 0.827399] pnp: PnP ACPI init [ 0.829707] pnp: PnP ACPI: found 6 devices [ 0.850870] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.853907] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.855855] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.857767] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.859933] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.862115] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.864823] NET: Registered protocol family 2 [ 0.867033] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.871685] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.874919] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.880854] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.885286] TCP: Hash tables configured (established 65536 bind 65536) [ 0.888691] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.892267] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.895349] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.898377] NET: Registered protocol family 1 [ 0.900928] RPC: Registered named UNIX socket transport module. [ 0.902868] RPC: Registered udp transport module. [ 0.904839] RPC: Registered tcp transport module. [ 0.906033] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.907747] NET: Registered protocol family 44 [ 0.909145] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.910983] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.912801] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.914804] PCI: CLS 0 bytes, default 64 [ 0.916242] Unpacking initramfs... [ 2.314769] debug: unmapping init [mem 0xffff9c9bfcc54000-0xffff9c9bfffbffff] [ 2.318409] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.320908] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.323831] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.836463] Initialise system trusted keyrings [ 2.838426] Key type blacklist registered [ 2.840491] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.849202] zbud: loaded [ 2.852529] *** VALIDATE nfs *** [ 2.854026] *** VALIDATE nfs4 *** [ 2.855941] pstore: using deflate compression [ 2.859383] Platform Keyring initialized [ 2.978684] NET: Registered protocol family 38 [ 2.981190] Key type asymmetric registered [ 2.982864] Asymmetric key parser 'x509' registered [ 2.985109] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.988688] io scheduler mq-deadline registered [ 2.990610] io scheduler kyber registered [ 2.992735] io scheduler bfq registered [ 2.995190] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.998391] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.001786] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.004752] ACPI: Power Button [PWRF] [ 3.010595] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.016798] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.031919] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.038182] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.055806] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.086656] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.122568] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.130634] Non-volatile memory driver v1.3 [ 3.132215] Linux agpgart interface v0.103 [ 3.169083] virtio_blk virtio1: [vda] 145808 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.172088] vda: detected capacity change from 0 to 74653696 [ 3.188495] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.191407] vdb: detected capacity change from 0 to 1073741824 [ 3.206648] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.209723] vdc: detected capacity change from 0 to 2621440000 [ 3.227635] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.230294] vdd: detected capacity change from 0 to 2621440000 [ 3.247753] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.250977] vde: detected capacity change from 0 to 4294967296 [ 3.266761] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.269781] vdf: detected capacity change from 0 to 4294967296 [ 3.277885] libphy: Fixed MDIO Bus: probed [ 3.283111] usbcore: registered new interface driver usbserial_generic [ 3.285525] usbserial: USB Serial support registered for generic [ 3.287942] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.292062] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.293641] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.295707] mousedev: PS/2 mouse device common for all mice [ 3.298544] rtc_cmos 00:05: RTC can wake from S4 [ 3.302715] rtc_cmos 00:05: registered as rtc0 [ 3.304489] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.307148] intel_pstate: CPU model not supported [ 3.310212] hid: raw HID events driver (C) Jiri Kosina [ 3.310683] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.312092] usbcore: registered new interface driver usbhid [ 3.317638] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.321087] usbhid: USB HID core driver [ 3.321273] drop_monitor: Initializing network drop monitor service [ 3.321400] Initializing XFRM netlink socket [ 3.328955] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.331812] NET: Registered protocol family 10 [ 3.338871] Segment Routing with IPv6 [ 3.340430] NET: Registered protocol family 17 [ 3.342766] mpls_gso: MPLS GSO support [ 3.348453] RAS: Correctable Errors collector initialized. [ 3.350784] AVX version of gcm_enc/dec engaged. [ 3.352531] AES CTR mode by8 optimization enabled [ 3.428254] sched_clock: Marking stable (3428232899, 0)->(4376565271, -948332372) [ 3.431531] registered taskstats version 1 [ 3.433863] Loading compiled-in X.509 certificates [ 3.435833] zswap: loaded using pool lzo/zbud [ 3.465146] Key type big_key registered [ 3.480298] Key type encrypted registered [ 3.481929] ima: No TPM chip found, activating TPM-bypass! [ 3.484526] ima: Allocated hash algorithm: sha1 [ 3.486113] ima: No architecture policies found [ 3.488299] evm: Initialising EVM extended attributes: [ 3.490079] evm: security.selinux [ 3.491321] evm: security.ima [ 3.492527] evm: security.capability [ 3.493814] evm: HMAC attrs: 0x1 [ 3.496289] rtc_cmos 00:05: setting system clock to 2026-07-20 21:47:10 UTC (1784584030) [ 3.502589] debug: unmapping init [mem 0xffffffff84e03000-0xffffffff84ffffff] [ 3.505692] debug: unmapping init [mem 0xffffffff83b82000-0xffffffff83e58fff] [ 3.513587] Write protecting the kernel read-only data: 28672k [ 3.516923] debug: unmapping init [mem 0xffffffff82203000-0xffffffff823fffff] [ 3.519402] debug: unmapping init [mem 0xffffffff82b14000-0xffffffff82bfffff] [ 3.550794] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.558403] systemd[1]: Detected virtualization kvm. [ 3.560828] systemd[1]: Detected architecture x86-64. [ 3.562568] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.585916] systemd[1]: No hostname configured. [ 3.587686] systemd[1]: Set hostname to . [ 3.589488] random: systemd: uninitialized urandom read (16 bytes read) [ 3.591541] systemd[1]: Initializing machine ID from random generator. [ 3.650858] random: ln: uninitialized urandom read (6 bytes read) [ 3.739267] random: systemd: uninitialized urandom read (16 bytes read) [ 3.741951] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.747173] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.751303] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.425529] device-mapper: uevent: version 1.0.3 [ 4.427455] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.273976] virtio_net virtio0 ens2: renamed from eth0 [ 5.297323] random: fast init done [ 5.364938] scsi host0: ata_piix [ 5.422212] scsi host1: ata_piix [ 5.424301] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.426907] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.218021] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.999186] random: crng init done [ 10.000468] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.533580] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.670889] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.925121] SELinux: Disabled at runtime. [ 11.983106] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.991408] systemd[1]: Detected virtualization kvm. [ 11.993095] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.528860] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.532194] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.538748] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.543992] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.547885] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.554979] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.561507] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.628534] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.963892] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.222096] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.239284] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.349997] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.364827] EDAC sbridge: Ver: 1.1.2 [ 15.176860] Key type dns_resolver registered [ 15.480241] NFS: Registering the id_resolver key type [ 15.482018] Key type id_resolver registered [ 15.483638] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg445-server login: [ 75.760868] libcfs: loading out-of-tree module taints kernel. [ 75.803231] Key type ._llcrypt registered [ 75.804858] Key type .llcrypt registered [ 75.932437] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 98.872802] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 102.342317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 102.355897] alg: No test for adler32 (adler32-zlib) [ 104.704204] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 106.142051] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 107.976594] Key type lgssc registered [ 110.028270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.074682] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 180.860605] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 196.248805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 196.316876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.994107] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 198.075687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 198.247882] Lustre: lustre-MDT0000: new disk, initializing [ 198.391229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.421752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 203.836532] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.261904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.368424] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 219.393304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 219.397290] Lustre: Skipped 1 previous similar message [ 219.510161] Lustre: lustre-MDT0001: new disk, initializing [ 219.577119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.600365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 219.617816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.161032] hrtimer: interrupt took 1725651 ns [ 224.448894] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.958350] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 240.388073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.692258] Lustre: lustre-OST0000: new disk, initializing [ 240.696295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.705276] Lustre: 8456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.830591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 247.851324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 247.870258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 247.964577] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.965808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 263.943966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.120177] Lustre: lustre-OST0001: new disk, initializing [ 264.127706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.144088] Lustre: 9531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.218617] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 270.603696] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 272.430714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 272.452153] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 272.488797] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 284.480918] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 291.952175] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 299.119441] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing check_logdir /tmp/testlogs/ [ 304.306758] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing yml_node [ 310.704132] Lustre: DEBUG MARKER: Client: 2.17.54.183 [ 314.304361] Lustre: DEBUG MARKER: MDS: 2.17.54.183 [ 317.730296] Lustre: DEBUG MARKER: OSS: 2.17.54.183 [ 319.917338] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 20 17:52:25 EDT 2026 [ 341.576048] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 343.317276] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 362.979486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 362.996310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 363.028807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.515187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.516880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 364.532311] Lustre: Skipped 1 previous similar message [ 364.545224] Lustre: Skipped 3 previous similar messages [ 365.940227] Lustre: server umount lustre-MDT0000 complete [ 370.333403] LustreError: 9530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584397 with bad export cookie 6738997596959968459 [ 370.339965] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 370.351339] LustreError: 9530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 374.753301] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.755763] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.758397] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.774315] Lustre: Skipped 2 previous similar messages [ 374.803430] LustreError: 6524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 375.779617] LustreError: 9255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 375.807805] LustreError: 9255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 376.920353] Lustre: server umount lustre-MDT0001 complete [ 380.827224] Lustre: server umount lustre-OST0000 complete [ 385.068879] Lustre: server umount lustre-OST0001 complete [ 392.864394] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 403.756554] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 445.752275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.340641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 446.397492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 446.572205] Lustre: lustre-MDT0000: new disk, initializing [ 446.782767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.820032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 453.793601] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 467.452070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.676426] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 467.817294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 467.827792] Lustre: Skipped 1 previous similar message [ 468.091402] Lustre: lustre-MDT0001: new disk, initializing [ 468.244353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 468.323723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 468.345192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 472.344179] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 481.437340] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 488.046455] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.114914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 496.669260] Lustre: lustre-OST0000: new disk, initializing [ 496.674320] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 496.680891] Lustre: 17058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 496.819441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.875084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 497.890727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 498.024637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 504.148038] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 515.560993] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 521.049357] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 521.329022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 526.980833] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 527.332144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 533.988526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 533.996912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.003237] LustreError: Skipped 1 previous similar message [ 534.008257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.008266] Lustre: Skipped 3 previous similar messages [ 534.063051] Lustre: Skipped 1 previous similar message [ 535.950712] Lustre: server umount lustre-OST0000 complete [ 544.227102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.235464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 544.243362] Lustre: Skipped 1 previous similar message [ 544.252750] Lustre: Skipped 1 previous similar message [ 546.586810] Lustre: server umount lustre-MDT0000 complete [ 549.350507] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.383937] LustreError: 15095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 550.771378] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584577 with bad export cookie 6738997596959969992 [ 550.778782] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.783734] LustreError: 15081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 551.126481] Lustre: server umount lustre-MDT0001 complete [ 555.368690] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 17:56:21 (1784584581) [ 563.067677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.629788] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 563.708317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.783842] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 568.802978] LustreError: 19323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.946614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.044135] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.064956] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 585.666567] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 594.255878] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 601.242995] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 609.519085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 615.892557] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 623.458983] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 644.068908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 644.072241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 644.109224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 644.128495] Lustre: Skipped 2 previous similar messages [ 648.547932] Lustre: server umount lustre-OST0000 complete [ 658.555766] Lustre: server umount lustre-MDT0000 complete [ 659.426811] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.470271] LustreError: 19318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 662.873393] LustreError: 20056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584689 with bad export cookie 6738997596959970874 [ 662.881153] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.887071] LustreError: 20056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 663.197159] Lustre: server umount lustre-MDT0001 complete [ 674.985753] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 677.304701] Key type lgssc unregistered [ 677.570695] LNet: 22877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.575831] LNetError: 22877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.602422] LNet: Removed LNI 192.168.204.145@tcp [ 678.356181] Key type .llcrypt unregistered [ 678.359843] Key type ._llcrypt unregistered [ 701.178358] Key type ._llcrypt registered [ 701.182586] Key type .llcrypt registered [ 701.330095] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 702.022861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 702.073358] alg: No test for adler32 (adler32-zlib) [ 703.265316] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 703.511882] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 705.232177] Key type lgssc registered [ 706.524953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 714.073843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 714.116179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.709075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.427974] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 725.990470] LustreError: 24293:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.019360] LustreError: 24293:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 729.740134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.289776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 735.132433] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 743.419312] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 751.188843] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 758.771319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 759.128919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.183637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 765.761915] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 774.007796] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 791.009291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 791.009582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.017845] LustreError: Skipped 1 previous similar message [ 791.042022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 791.045108] Lustre: Skipped 1 previous similar message [ 791.064719] Lustre: Skipped 1 previous similar message [ 793.371366] Lustre: server umount lustre-OST0000 complete [ 801.250586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 801.273822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 801.289296] Lustre: Skipped 1 previous similar message [ 804.041428] Lustre: server umount lustre-MDT0000 complete [ 806.371142] LustreError: 24289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 806.394243] LustreError: 24289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 807.898229] LustreError: 24273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584834 with bad export cookie 7501938689852473937 [ 807.932908] LustreError: 24273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 807.944060] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 808.316535] Lustre: server umount lustre-MDT0001 complete [ 820.143555] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 822.873352] Key type lgssc unregistered [ 823.224366] LNet: 28268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.233818] LNetError: 28268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.264797] LNet: Removed LNI 192.168.204.145@tcp [ 824.241070] Key type .llcrypt unregistered [ 824.244067] Key type ._llcrypt unregistered [ 848.865421] Key type ._llcrypt registered [ 848.871239] Key type .llcrypt registered [ 849.041203] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 850.191177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.229187] alg: No test for adler32 (adler32-zlib) [ 851.402587] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 851.649660] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 853.344584] Key type lgssc registered [ 854.419657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 862.152351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 862.177461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.687415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.748292] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 873.955360] LustreError: 29694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.981710] LustreError: 29694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 877.006220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.293829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 880.968330] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 891.138813] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 898.461311] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 913.358603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.696788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.108075] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 927.830713] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 933.179450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 943.588103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.594550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 943.603815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 943.614964] Lustre: Skipped 1 previous similar message [ 946.687864] Lustre: server umount lustre-OST0000 complete [ 953.828625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.836300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 953.842507] Lustre: Skipped 2 previous similar messages [ 956.420252] Lustre: server umount lustre-MDT0000 complete [ 958.949023] LustreError: 29688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.971145] LustreError: 29688:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 960.023894] LustreError: 29674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584987 with bad export cookie 2707029308385696286 [ 960.026949] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 960.037881] LustreError: 29674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 960.307392] Lustre: server umount lustre-MDT0001 complete [ 970.079398] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 972.748429] Key type lgssc unregistered [ 972.970697] LNet: 33596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.977880] LNetError: 33596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.993895] LNet: Removed LNI 192.168.204.145@tcp [ 973.684558] Key type .llcrypt unregistered [ 973.686951] Key type ._llcrypt unregistered [ 994.854871] Key type ._llcrypt registered [ 994.859782] Key type .llcrypt registered [ 994.993934] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 995.845416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 995.875123] alg: No test for adler32 (adler32-zlib) [ 996.872923] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 997.131035] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 998.760353] Key type lgssc registered [ 999.840805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1007.110224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1007.164757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.654203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.954291] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1018.859588] LustreError: 35012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.888263] LustreError: 35012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1022.343142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.855088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1027.584585] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1035.503683] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1042.035666] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1050.153277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1050.692126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1053.739252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1057.228721] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1065.525378] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1077.223566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1077.233802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1077.265877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1082.340571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1082.349650] Lustre: Skipped 2 previous similar messages [ 1082.709141] Lustre: server umount lustre-OST0000 complete [ 1089.509096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1089.522850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.537042] Lustre: Skipped 1 previous similar message [ 1089.544697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1092.578298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1092.578596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1092.592327] Lustre: Skipped 1 previous similar message [ 1097.697995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.717675] Lustre: Skipped 1 previous similar message [ 1102.304860] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1102.746902] Lustre: server umount lustre-MDT0000 complete [ 1102.817723] LustreError: 35012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.858970] LustreError: 35012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1107.071472] LustreError: 34991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585134 with bad export cookie 5654516411574391465 [ 1107.084203] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1107.363177] Lustre: server umount lustre-MDT0001 complete [ 1121.468612] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1124.935799] Key type lgssc unregistered [ 1125.324763] LNet: 38547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.345139] LNetError: 38547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.369539] LNet: Removed LNI 192.168.204.145@tcp [ 1126.470168] Key type .llcrypt unregistered [ 1126.474331] Key type ._llcrypt unregistered [ 1151.867535] Key type ._llcrypt registered [ 1151.871397] Key type .llcrypt registered [ 1152.004574] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1153.794203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1153.838415] alg: No test for adler32 (adler32-zlib) [ 1155.513833] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1156.021819] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1157.736194] Key type lgssc registered [ 1159.382965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1169.129981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1169.142824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.835674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1175.461768] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1180.133860] LustreError: 39960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.172892] LustreError: 39960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1184.972049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.254575] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1185.364927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1190.919799] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1199.912974] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1206.178989] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1213.639983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1214.113073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.152587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1222.059931] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1231.058303] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1238.004440] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1241.060169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1241.064561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.082088] LustreError: Skipped 1 previous similar message [ 1241.087112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.097184] Lustre: Skipped 1 previous similar message [ 1241.129936] Lustre: Skipped 1 previous similar message [ 1243.094079] Lustre: lustre-OST0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1243.543549] Lustre: server umount lustre-OST0000 complete [ 1266.657618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.661092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.670089] Lustre: Skipped 1 previous similar message [ 1266.684803] Lustre: Skipped 1 previous similar message [ 1270.602284] Lustre: server umount lustre-MDT0000 complete [ 1271.791111] LustreError: 39960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.826571] LustreError: 39960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1274.407950] LustreError: 39942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585301 with bad export cookie 11511883224606124725 [ 1274.410469] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1274.420646] LustreError: 39942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1280.846141] Lustre: server umount lustre-MDT0001 complete [ 1293.846423] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1296.121298] Key type lgssc unregistered [ 1296.387500] LNet: 43675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.393123] LNetError: 43675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.411184] LNet: Removed LNI 192.168.204.145@tcp [ 1297.149062] Key type .llcrypt unregistered [ 1297.151904] Key type ._llcrypt unregistered [ 1323.226930] Key type ._llcrypt registered [ 1323.230161] Key type .llcrypt registered [ 1323.329649] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1324.191320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.223186] alg: No test for adler32 (adler32-zlib) [ 1325.212132] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1325.411654] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1327.056351] Key type lgssc registered [ 1327.959288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1335.764657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1335.786749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.315176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.818275] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1347.554207] LustreError: 45088:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.576578] LustreError: 45088:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1349.928605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.190056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1354.474770] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1362.411605] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1368.815675] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1375.748764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1376.154395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1382.169883] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1382.334909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1389.631076] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1400.819606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.820303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1400.841648] Lustre: Skipped 2 previous similar messages [ 1400.848689] Lustre: Skipped 1 previous similar message [ 1402.363164] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.45@tcp (stopping) [ 1402.367415] Lustre: Skipped 1 previous similar message [ 1405.922957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1406.209523] Lustre: server umount lustre-MDT0000 complete [ 1407.436130] LustreError: 45083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1410.453135] LustreError: 45067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585437 with bad export cookie 12793759827763060039 [ 1410.454046] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1410.468886] LustreError: 45067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1410.505944] LustreError: 45082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1410.557611] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.45@tcp (stopping) [ 1410.574285] Lustre: Skipped 2 previous similar messages [ 1411.041657] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.077080] LustreError: 45084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.077179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1415.105925] LustreError: 45084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1415.130365] Lustre: Skipped 1 previous similar message [ 1417.035034] Lustre: server umount lustre-MDT0001 complete [ 1446.368134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1446.458967] Lustre: server umount lustre-OST0000 complete [ 1462.435396] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1464.779331] Key type lgssc unregistered [ 1465.047707] LNet: 48952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.060529] LNetError: 48952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.077093] LNet: Removed LNI 192.168.204.145@tcp [ 1465.787939] Key type .llcrypt unregistered [ 1465.792867] Key type ._llcrypt unregistered [ 1486.593983] Key type ._llcrypt registered [ 1486.596507] Key type .llcrypt registered [ 1486.671887] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1487.486117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1487.606687] alg: No test for adler32 (adler32-zlib) [ 1488.661090] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1488.976613] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1490.656179] Key type lgssc registered [ 1491.865879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1499.330387] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1499.355029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1514.912511] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1520.033723] LustreError: 50319:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 1546.786242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1553.331358] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1561.514457] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1729.863770] Lustre: server umount lustre-OST0000 complete [ 1747.146849] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1749.748200] Key type lgssc unregistered [ 1749.991327] LNet: 52004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.997963] LNetError: 52004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1750.024570] LNet: Removed LNI 192.168.204.145@tcp [ 1750.739932] Key type .llcrypt unregistered [ 1750.746240] Key type ._llcrypt unregistered [ 1774.473957] Key type ._llcrypt registered [ 1774.477156] Key type .llcrypt registered [ 1774.570534] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1775.461488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1775.516793] alg: No test for adler32 (adler32-zlib) [ 1776.696937] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1776.965038] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1778.640774] Key type lgssc registered [ 1779.735579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.415225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1787.450234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1788.923366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.360418] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1798.115693] LustreError: 53418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1798.126269] LustreError: 53418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1802.448469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1802.915369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1807.745117] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1815.898917] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1822.001708] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1829.106814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1829.517359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.668873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1835.758505] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1843.777398] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1847.205843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.213887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1852.322078] Lustre: server umount lustre-OST0000 complete [ 1857.508198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.522899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1857.536458] Lustre: Skipped 2 previous similar messages [ 1862.636136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.646514] Lustre: Skipped 1 previous similar message [ 1867.747621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1870.304380] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1870.539651] Lustre: server umount lustre-MDT0000 complete [ 1872.870147] LustreError: 54148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.898724] LustreError: 54148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1874.349849] LustreError: 53398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585901 with bad export cookie 16497044753599576657 [ 1874.353420] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1874.363209] LustreError: 53398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1874.762564] Lustre: server umount lustre-MDT0001 complete [ 1885.464928] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 1888.341948] Key type lgssc unregistered [ 1888.658061] LNet: 56951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.667586] LNetError: 56951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.693873] LNet: Removed LNI 192.168.204.145@tcp [ 1889.609634] Key type .llcrypt unregistered [ 1889.613694] Key type ._llcrypt unregistered [ 1914.785141] Key type ._llcrypt registered [ 1914.787959] Key type .llcrypt registered [ 1914.883260] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 1915.979962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.065616] alg: No test for adler32 (adler32-zlib) [ 1917.213719] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1917.488245] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 1919.168290] Key type lgssc registered [ 1920.494916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1928.016164] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1928.054164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1943.712614] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1948.832399] LustreError: 58340:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 1975.520826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.193706] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 1990.070313] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1997.190039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.750286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.822531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2001.960579] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2002.918311] LustreError: 58366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2002.936536] LustreError: 58366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2008.051978] LustreError: 58365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.089221] LustreError: 58366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.113969] LustreError: 58366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2011.041810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.508972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2016.866474] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2025.517146] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2031.881041] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2037.241685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2037.242514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.257591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2037.276661] Lustre: Skipped 1 previous similar message [ 2040.284329] Lustre: server umount lustre-OST0000 complete [ 2062.808682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.827902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2062.837355] Lustre: Skipped 2 previous similar messages [ 2067.938189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2067.947141] Lustre: Skipped 1 previous similar message [ 2073.062101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.152366] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2077.165477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2077.186090] Lustre: Skipped 2 previous similar messages [ 2077.351670] Lustre: server umount lustre-MDT0000 complete [ 2081.330990] LustreError: 58348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586108 with bad export cookie 12237086319143139445 [ 2081.337353] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2081.341470] LustreError: 58348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2088.055140] Lustre: server umount lustre-MDT0001 complete [ 2098.644566] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2101.480084] Key type lgssc unregistered [ 2101.787922] LNet: 62086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.791473] LNetError: 62086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.812895] LNet: Removed LNI 192.168.204.145@tcp [ 2102.681759] Key type .llcrypt unregistered [ 2102.687982] Key type ._llcrypt unregistered [ 2127.750897] Key type ._llcrypt registered [ 2127.755819] Key type .llcrypt registered [ 2127.946939] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2129.060566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2129.144037] alg: No test for adler32 (adler32-zlib) [ 2130.392601] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2130.604674] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2132.304169] Key type lgssc registered [ 2133.484299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2140.712978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2140.748500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.170396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.729297] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2152.418383] LustreError: 63498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.429087] LustreError: 63498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2154.235174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.480739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2158.807501] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2167.795701] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2174.389409] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2181.410884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2181.864827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.595689] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2188.014937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2194.052612] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2213.862100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2213.879207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.891387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.889862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2214.892313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2214.910599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.284087] Lustre: server umount lustre-OST0000 complete [ 2220.005211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2220.009687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2220.022173] Lustre: Skipped 1 previous similar message [ 2220.033943] Lustre: Skipped 2 previous similar messages [ 2225.121448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2225.134665] Lustre: Skipped 1 previous similar message [ 2230.258048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2230.271909] Lustre: Skipped 1 previous similar message [ 2234.337034] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2234.517296] Lustre: server umount lustre-MDT0000 complete [ 2235.367100] LustreError: 64718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.385888] LustreError: 64718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2238.088241] LustreError: 63478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586265 with bad export cookie 13378794682079335902 [ 2238.093293] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.093610] LustreError: 63478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2238.492103] Lustre: server umount lustre-MDT0001 complete [ 2249.060520] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2251.155555] Key type lgssc unregistered [ 2251.401836] LNet: 67052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.425698] LNetError: 67052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.437046] LNet: Removed LNI 192.168.204.145@tcp [ 2252.189809] Key type .llcrypt unregistered [ 2252.194300] Key type ._llcrypt unregistered [ 2280.122340] Key type ._llcrypt registered [ 2280.132117] Key type .llcrypt registered [ 2280.262347] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2281.382262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2281.458082] alg: No test for adler32 (adler32-zlib) [ 2282.517646] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2282.814806] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2284.504388] Key type lgssc registered [ 2285.665368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2293.517627] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 18:25:19 (1784586319) [ 2299.647861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2299.688865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2301.202972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2305.506702] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2311.654104] LustreError: 68757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2311.676112] LustreError: 68757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2313.705526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.009250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2317.568870] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2324.582554] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2330.711183] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2339.075856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2339.437908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2346.603646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2346.693088] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2355.306906] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2364.332674] Lustre: Failing over lustre-MDT0000 [ 2364.398904] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.413277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2364.430167] Lustre: Skipped 2 previous similar messages [ 2364.859600] Lustre: server umount lustre-MDT0000 complete [ 2367.467578] LustreError: 68752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.509911] LustreError: 70245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.545959] LustreError: 70245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2372.561439] LustreError: 68753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.230680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.427358] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2376.525442] Lustre: *** cfs_fail_loc=135, val=0*** [ 2376.622517] LustreError: 71844:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2376.628691] LustreError: 71844:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.145@tcp: cfg command failed: rc = -2 [ 2376.636992] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2376.636992] [ 2376.660100] LustreError: MGC192.168.204.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2376.670983] LustreError: 71832:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2376.686361] LustreError: 71832:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2376.698290] LustreError: 71832:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2376.706412] LustreError: 71832:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2376.752036] Lustre: server umount lustre-MDT0000 complete [ 2376.757342] LustreError: 71832:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2377.682695] LustreError: 68751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.706194] LustreError: 68751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2383.961556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.075266] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2384.092827] LustreError: 68748:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2384.321194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.384950] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2387.915986] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2389.355345] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2389.498456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2389.587112] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2389.653922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2399.721109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2399.723209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.735052] Lustre: Skipped 3 previous similar messages [ 2403.579836] Lustre: server umount lustre-OST0000 complete [ 2409.961661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.987364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.993935] Lustre: Skipped 2 previous similar messages [ 2414.000315] Lustre: server umount lustre-MDT0000 complete [ 2415.082414] LustreError: 68756:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.107056] LustreError: 68756:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2417.543311] LustreError: 68736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586444 with bad export cookie 8101510158390974767 [ 2417.553620] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2417.558319] LustreError: 68736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2417.898481] Lustre: server umount lustre-MDT0001 complete [ 2429.886705] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2432.942838] Key type lgssc unregistered [ 2433.388503] LNet: 73626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.397852] LNetError: 73626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2433.412351] LNet: Removed LNI 192.168.204.145@tcp [ 2434.248249] Key type .llcrypt unregistered [ 2434.250647] Key type ._llcrypt unregistered [ 2461.455490] Key type ._llcrypt registered [ 2461.458188] Key type .llcrypt registered [ 2461.570800] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2462.993337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2463.016671] alg: No test for adler32 (adler32-zlib) [ 2464.151394] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2464.415768] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2466.216171] Key type lgssc registered [ 2467.371781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2475.921244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2475.961300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.331522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.136878] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2487.782552] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.801672] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2490.835435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.352111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.023292] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2503.963473] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2509.934052] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2516.118636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2516.458040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.565495] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2522.620495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2530.155907] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2536.514445] Lustre: Failing over lustre-MDT0000 [ 2536.682264] Lustre: server umount lustre-MDT0000 complete [ 2537.442162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2537.450093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.469107] LustreError: 77439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.486271] LustreError: 77439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2537.957749] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.537654] LustreError: 75035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.550341] LustreError: 75035:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2544.995301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.195205] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2545.366293] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2545.545523] LustreError: 78126:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2545.551743] LustreError: 78126:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.145@tcp: cfg command failed: rc = -9 [ 2545.557338] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2545.557338] [ 2545.566709] LustreError: MGC192.168.204.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2545.584077] LustreError: 78114:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2545.589142] LustreError: 78114:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2545.604929] LustreError: 78114:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2545.610273] LustreError: 78114:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2545.638820] Lustre: server umount lustre-MDT0000 complete [ 2545.641037] LustreError: 78114:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2547.663866] LustreError: 75034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.681326] LustreError: 75034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2551.779882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.893142] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2552.119778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.180889] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2552.775566] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2556.172481] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2557.422860] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2557.453480] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2557.512194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2567.652082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.658525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2567.664102] Lustre: Skipped 2 previous similar messages [ 2567.674904] Lustre: Skipped 1 previous similar message [ 2572.778935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2572.786857] Lustre: Skipped 1 previous similar message [ 2577.889678] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2577.899390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2577.915657] Lustre: Skipped 1 previous similar message [ 2578.031728] Lustre: server umount lustre-OST0000 complete [ 2583.016394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2583.019060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2583.029653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.060566] Lustre: Skipped 1 previous similar message [ 2583.075245] Lustre: Skipped 1 previous similar message [ 2587.838589] Lustre: server umount lustre-MDT0000 complete [ 2588.130921] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.142572] LustreError: 75040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2591.037096] LustreError: 75019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586618 with bad export cookie 6521598657207785934 [ 2591.045060] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2591.049074] LustreError: 75019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2591.377360] Lustre: server umount lustre-MDT0001 complete [ 2603.030370] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2605.864585] Key type lgssc unregistered [ 2606.319654] LNet: 79908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2606.326872] LNetError: 79908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2606.345615] LNet: Removed LNI 192.168.204.145@tcp [ 2607.266868] Key type .llcrypt unregistered [ 2607.268529] Key type ._llcrypt unregistered [ 2623.382755] Key type ._llcrypt registered [ 2623.387212] Key type .llcrypt registered [ 2623.514737] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2623.853428] Key type .llcrypt unregistered [ 2623.855943] Key type ._llcrypt unregistered [ 2651.699860] Key type ._llcrypt registered [ 2651.701419] Key type .llcrypt registered [ 2651.824091] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2653.480678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2653.551858] alg: No test for adler32 (adler32-zlib) [ 2654.754692] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2655.042559] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2656.826096] Key type lgssc registered [ 2658.338552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2665.241914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2665.263126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.729773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2670.786403] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2676.205305] LustreError: 81811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.219868] LustreError: 81811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2679.030943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.376437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2683.432482] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2690.912082] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2696.694129] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2702.936427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2703.237467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.193221] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2711.495547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2717.818964] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2729.953490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2729.961675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.979371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2732.014814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.018172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2732.032070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.055103] Lustre: Skipped 1 previous similar message [ 2734.399598] Lustre: server umount lustre-OST0000 complete [ 2740.707623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2740.715127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.741150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2744.354756] Lustre: server umount lustre-MDT0000 complete [ 2747.361736] LustreError: 83423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.377320] LustreError: 83423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2748.062438] LustreError: 81791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586775 with bad export cookie 16147991669634321379 [ 2748.063316] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2748.076384] LustreError: 81791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2748.459707] Lustre: server umount lustre-MDT0001 complete [ 2759.207867] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2762.115380] Key type lgssc unregistered [ 2762.379904] LNet: 85346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2762.390117] LNetError: 85346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2762.413846] LNet: Removed LNI 192.168.204.145@tcp [ 2763.381516] Key type .llcrypt unregistered [ 2763.385819] Key type ._llcrypt unregistered [ 2787.930324] Key type ._llcrypt registered [ 2787.936773] Key type .llcrypt registered [ 2788.077028] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2789.584420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2789.643143] alg: No test for adler32 (adler32-zlib) [ 2790.794320] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2791.097994] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2792.864180] Key type lgssc registered [ 2794.319115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2803.408525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2803.438837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2804.923262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2809.872709] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2814.433847] LustreError: 86760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.451286] LustreError: 86760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2819.561553] LustreError: 86761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.305088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.883725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2826.751820] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2836.258701] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2843.599885] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2851.976560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2852.329758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.019725] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2869.622102] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2874.875420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2885.116937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2885.122921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.143534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2885.151069] Lustre: Skipped 1 previous similar message [ 2887.412274] Lustre: server umount lustre-OST0000 complete [ 2892.774907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2892.793838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2892.819656] Lustre: Skipped 1 previous similar message [ 2892.835785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.335323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.335960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.341102] Lustre: server umount lustre-MDT0000 complete [ 2900.455420] LustreError: 86755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2900.467638] LustreError: 86755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2903.021295] LustreError: 86742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586930 with bad export cookie 9631132663666483774 [ 2903.025187] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2903.032690] LustreError: 86742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2903.434770] Lustre: server umount lustre-MDT0001 complete [ 2914.853591] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 2918.067672] Key type lgssc unregistered [ 2918.429603] LNet: 90295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2918.453499] LNetError: 90295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2918.474567] LNet: Removed LNI 192.168.204.145@tcp [ 2919.616259] Key type .llcrypt unregistered [ 2919.619866] Key type ._llcrypt unregistered [ 2945.206214] Key type ._llcrypt registered [ 2945.208659] Key type .llcrypt registered [ 2945.321579] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 2946.431813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2946.497080] alg: No test for adler32 (adler32-zlib) [ 2947.587305] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2947.851415] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 2949.512266] Key type lgssc registered [ 2950.547044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2958.151886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2958.179093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2959.609561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.046250] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2969.066797] LustreError: 91714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2969.099531] LustreError: 91714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2972.870042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.089435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2977.294341] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 2986.667861] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2992.524800] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2999.565147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.943767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3004.034836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3006.240307] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3013.770415] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3034.084308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.084856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3034.108702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3034.133702] Lustre: Skipped 1 previous similar message [ 3036.651354] Lustre: server umount lustre-OST0000 complete [ 3044.331757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.337624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3044.347317] Lustre: Skipped 2 previous similar messages [ 3046.866722] Lustre: server umount lustre-MDT0000 complete [ 3049.441396] LustreError: 92746:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.451895] LustreError: 92746:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3050.301729] LustreError: 91694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587077 with bad export cookie 6987185183529379496 [ 3050.307650] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3050.316877] LustreError: 91694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3050.660560] Lustre: server umount lustre-MDT0001 complete [ 3064.624995] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3067.669920] Key type lgssc unregistered [ 3067.972832] LNet: 95273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3067.980796] LNetError: 95273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3067.998189] LNet: Removed LNI 192.168.204.145@tcp [ 3069.088966] Key type .llcrypt unregistered [ 3069.090622] Key type ._llcrypt unregistered [ 3094.653570] Key type ._llcrypt registered [ 3094.656515] Key type .llcrypt registered [ 3094.755365] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3095.398154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3095.431927] alg: No test for adler32 (adler32-zlib) [ 3096.511881] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 3096.735582] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3098.432181] Key type lgssc registered [ 3099.316078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3107.176882] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3107.188454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.784856] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3127.905958] LustreError: 96658:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 3154.656273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3160.995870] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3169.939650] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3176.479017] Lustre: server umount lustre-OST0000 complete [ 3184.837898] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 18:40:11 (1784587211) [ 3191.312474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3191.957180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3196.332227] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3197.414368] LustreError: 98297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.428088] LustreError: 98297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3201.506936] LustreError: 98298:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.392056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3204.863700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3209.548580] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3217.497164] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3223.621217] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3230.572563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3230.942894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3236.105318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3236.565816] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3243.583593] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3259.885073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.888139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3259.899862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3264.044228] Lustre: server umount lustre-OST0000 complete [ 3270.125043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.147055] Lustre: Skipped 1 previous similar message [ 3270.153449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3270.167148] Lustre: Skipped 2 previous similar messages [ 3275.233811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3275.246169] Lustre: Skipped 1 previous similar message [ 3280.356196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3280.366061] Lustre: Skipped 1 previous similar message [ 3282.400208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3282.699516] Lustre: server umount lustre-MDT0000 complete [ 3285.474184] LustreError: 98297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3285.499861] LustreError: 98297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3286.542600] LustreError: 98278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587313 with bad export cookie 7989773389012815092 [ 3286.548981] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3286.554361] LustreError: 98278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3286.835112] Lustre: server umount lustre-MDT0001 complete [ 3298.107117] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3300.539671] Key type lgssc unregistered [ 3300.811857] LNet: 101990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3300.822531] LNetError: 101990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3301.863508] LNet: Removed LNI 192.168.204.145@tcp [ 3302.667783] Key type .llcrypt unregistered [ 3302.671837] Key type ._llcrypt unregistered [ 3346.373355] Key type ._llcrypt registered [ 3346.375602] Key type .llcrypt registered [ 3346.455845] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3347.466551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3347.540973] alg: No test for adler32 (adler32-zlib) [ 3348.664813] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 3348.972454] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3350.648315] Key type lgssc registered [ 3351.916189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3359.724752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3359.780412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.291809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.730562] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3371.500136] LustreError: 104179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3371.529729] LustreError: 104179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3374.177231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.573618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3378.526156] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3385.458970] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3391.596880] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3398.668103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3399.135188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3405.513662] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3406.338282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3413.511371] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3424.738407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3424.750635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.790540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3426.788049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.791147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3426.806356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3426.811189] Lustre: Skipped 1 previous similar message [ 3429.256092] Lustre: server umount lustre-OST0000 complete [ 3436.004545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3436.024620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.062667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3439.355113] Lustre: server umount lustre-MDT0000 complete [ 3442.150971] LustreError: 104174:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.194860] LustreError: 104174:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3443.045941] LustreError: 104161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587470 with bad export cookie 15852761128914446629 [ 3443.046250] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3443.060894] LustreError: 104161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3443.588306] Lustre: server umount lustre-MDT0001 complete [ 3454.033317] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 3456.485753] Key type lgssc unregistered [ 3456.735890] LNet: 107713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3456.744148] LNetError: 107713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3456.765950] LNet: Removed LNI 192.168.204.145@tcp [ 3457.472538] Key type .llcrypt unregistered [ 3457.475202] Key type ._llcrypt unregistered [ 3472.767139] Key type ._llcrypt registered [ 3472.773723] Key type .llcrypt registered [ 3472.898154] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3473.963633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3473.985296] alg: No test for adler32 (adler32-zlib) [ 3475.019410] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 3475.290523] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 3477.018265] Key type lgssc registered [ 3477.933913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3485.380377] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3485.403117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3501.024375] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3506.144550] LustreError: 108935:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 3532.768209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3539.589100] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3548.662761] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3556.091803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.432180] LustreError: MGC192.168.204.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3556.453935] LustreError: 109959:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3556.465992] LustreError: 109959:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3556.492717] LustreError: 109959:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3556.584171] Lustre: server umount lustre-MDT0000 complete [ 3556.588205] LustreError: 109959:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3576.800397] Lustre: 108273:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784587587/real 1784587587] req@00000000a010972c x1871275624500608/t0(0) o400->MGC192.168.204.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784587603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3576.847993] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3576.879173] LustreError: 108947:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3577.079434] Lustre: server umount lustre-OST0000 complete [ 3587.197133] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 3597.145996] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3631.667886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3631.908669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3631.931791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3632.006169] Lustre: lustre-MDT0000: new disk, initializing [ 3632.061925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.081594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3636.898716] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3648.473796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.558868] Lustre: 112751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3648.603782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3648.608522] Lustre: Skipped 1 previous similar message [ 3648.724925] Lustre: lustre-MDT0001: new disk, initializing [ 3648.867916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3648.927893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3648.944517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3654.464970] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3666.362869] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3673.299888] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3680.956854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3681.249990] Lustre: lustre-OST0000: new disk, initializing [ 3681.256875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3681.277382] Lustre: 114720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3681.406139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3688.580216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3688.592631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3688.775764] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3688.797149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3699.460867] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3704.179459] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3704.379096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3708.710964] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3708.902856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3714.534279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3714.537931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3714.561319] LustreError: Skipped 1 previous similar message [ 3714.562768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3714.578175] Lustre: Skipped 1 previous similar message [ 3716.949072] Lustre: server umount lustre-OST0000 complete [ 3720.674310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3720.682401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3720.698825] Lustre: Skipped 1 previous similar message [ 3720.707322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3724.774561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3724.774670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.514610] Lustre: server umount lustre-MDT0000 complete [ 3729.958729] LustreError: 112743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587756 with bad export cookie 33843316108378788 [ 3729.978680] LustreError: 112743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3729.980166] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3730.367594] Lustre: server umount lustre-MDT0001 complete [ 3738.852919] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 18:49:25 (1784587765) [ 3741.981455] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3756.200476] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 3764.272917] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 3801.093601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.367962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3801.407673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3801.497239] Lustre: lustre-MDT0000: new disk, initializing [ 3801.591732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.614896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3807.106700] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3817.649853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3817.754876] Lustre: 119621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3817.797269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3817.800396] Lustre: Skipped 1 previous similar message [ 3817.875249] Lustre: lustre-MDT0001: new disk, initializing [ 3817.954436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3817.991832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3818.004971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3822.444790] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3832.285475] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3838.193267] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3844.528457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3844.846900] Lustre: lustre-OST0000: new disk, initializing [ 3844.850657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3844.855694] Lustre: 121589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3844.939603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3851.032853] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3852.431563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3852.444743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3852.563952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3861.376925] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3865.527738] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3865.694370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3869.834980] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3870.037476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3873.251502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.252508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3873.255815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3873.255824] Lustre: Skipped 1 previous similar message [ 3873.259572] Lustre: Skipped 1 previous similar message [ 3873.262700] LustreError: Skipped 1 previous similar message [ 3878.177380] Lustre: server umount lustre-OST0000 complete [ 3883.493033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3883.496121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3883.513429] Lustre: Skipped 1 previous similar message [ 3883.528229] Lustre: Skipped 2 previous similar messages [ 3888.458315] Lustre: server umount lustre-MDT0000 complete [ 3888.611522] LustreError: 120846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3888.631588] LustreError: 120846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3892.241517] LustreError: 119614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587919 with bad export cookie 33843316108379670 [ 3892.243896] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3892.259486] LustreError: 119614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3892.816659] Lustre: server umount lustre-MDT0001 complete [ 3901.155243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3901.588994] LustreError: 123805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.653391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.845692] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3907.057751] LustreError: 123806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3911.146220] LustreError: 123805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3915.451343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3920.678336] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3928.416124] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3934.570271] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3941.331836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3948.535839] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 3957.534982] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3967.544950] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3973.601916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3973.619760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.640129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3978.628830] Lustre: server umount lustre-OST0000 complete [ 3989.010607] Lustre: server umount lustre-MDT0000 complete [ 3991.025639] LustreError: 123800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.068876] LustreError: 123800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3993.506417] LustreError: 123787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588020 with bad export cookie 33843316108380552 [ 3993.507748] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3993.540452] LustreError: 123787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3993.950971] Lustre: server umount lustre-MDT0001 complete [ 4006.650600] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 4010.322949] Key type lgssc unregistered [ 4010.722620] LNet: 127438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4010.728357] LNetError: 127438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4010.750274] LNet: Removed LNI 192.168.204.145@tcp [ 4011.842957] Key type .llcrypt unregistered [ 4011.845483] Key type ._llcrypt unregistered [ 4032.167674] Key type ._llcrypt registered [ 4032.170752] Key type .llcrypt registered [ 4032.329096] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 4050.317110] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 4051.510840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4051.540506] alg: No test for adler32 (adler32-zlib) [ 4052.554641] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 4052.792688] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 4054.488167] Key type lgssc registered [ 4055.709048] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4093.408418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4093.454898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.716590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4094.744108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4094.826292] Lustre: lustre-MDT0000: new disk, initializing [ 4094.946459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.971505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4100.878984] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4112.659892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4112.794374] Lustre: 130590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4112.905075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4112.912321] Lustre: Skipped 1 previous similar message [ 4113.039057] Lustre: lustre-MDT0001: new disk, initializing [ 4113.193125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4113.258119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4113.277912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4118.868080] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4129.374459] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4135.194256] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4141.556799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4141.892271] Lustre: lustre-OST0000: new disk, initializing [ 4141.895747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4141.904607] Lustre: 132563:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4142.018809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.698596] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4149.312628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4149.326158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4149.517815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4159.618049] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4164.647041] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4164.994670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4170.233781] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4170.551567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4175.336943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4175.337263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.350429] LustreError: Skipped 1 previous similar message [ 4175.357814] Lustre: Skipped 1 previous similar message [ 4175.360953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4179.652710] Lustre: server umount lustre-OST0000 complete [ 4185.058084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4185.072422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.096383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4185.099645] Lustre: Skipped 1 previous similar message [ 4190.927744] Lustre: server umount lustre-MDT0000 complete [ 4195.021318] LustreError: 130583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588222 with bad export cookie 11351007485255356300 [ 4195.032000] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4195.036083] LustreError: 130583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4195.428845] Lustre: server umount lustre-MDT0001 complete [ 4204.359293] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 18:57:10 (1784588230) [ 4210.963709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4211.330987] LustreError: 134975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4211.348151] LustreError: 134975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4211.398213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.751487] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4216.802180] LustreError: 134976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.927133] LustreError: 134975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.162522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4224.560133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4229.127791] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4237.488447] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4244.425227] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4247.014995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4247.024390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4247.036313] Lustre: Skipped 2 previous similar messages [ 4247.052245] Lustre: Skipped 2 previous similar messages [ 4249.122219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4249.131589] Lustre: Skipped 1 previous similar message [ 4253.195899] Lustre: server umount lustre-MDT0000 complete [ 4254.194890] LustreError: 134976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.226914] LustreError: 134976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4257.351540] LustreError: 134956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588284 with bad export cookie 11351007485255357182 [ 4257.374347] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4257.766790] Lustre: server umount lustre-MDT0001 complete [ 4266.685847] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 18:58:13 (1784588293) [ 4274.749484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4290.465683] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4295.652719] LustreError: 137391:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 4321.440248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.615212] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4340.233199] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4343.622659] Lustre: server umount lustre-OST0000 complete [ 4356.514625] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 18:59:42 (1784588382) [ 4364.922897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.303587] LustreError: 138839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.407390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.363751] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4375.524338] LustreError: 138840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.571156] LustreError: 138840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4381.046732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.869922] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4395.449860] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4402.561562] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4410.786487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.231899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.237149] Lustre: Skipped 1 previous similar message [ 4418.534673] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4427.606552] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4446.177974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.186814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4446.195191] Lustre: Skipped 2 previous similar messages [ 4446.213236] Lustre: Skipped 2 previous similar messages [ 4452.353806] Lustre: server umount lustre-MDT0000 complete [ 4456.420653] LustreError: 138839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4456.436367] LustreError: 138839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4456.599597] LustreError: 138819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588483 with bad export cookie 11351007485255357847 [ 4456.603455] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4456.611645] LustreError: 138819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4457.128837] Lustre: server umount lustre-MDT0001 complete [ 4471.770431] Lustre: server umount lustre-OST0000 complete [ 4481.151600] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 19:01:47 (1784588507) [ 4488.159747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4488.743944] LustreError: 142526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4488.808591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.501719] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4503.195690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.526621] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4503.538424] Lustre: Skipped 3 previous similar messages [ 4508.846511] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4516.886594] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4523.549780] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4531.783371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4533.202949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4539.621675] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4548.600452] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4553.459063] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4553.753257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4558.901811] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4559.164806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4563.952185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4563.952248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.960491] LustreError: Skipped 1 previous similar message [ 4563.961053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4563.974931] Lustre: Skipped 1 previous similar message [ 4567.348404] Lustre: server umount lustre-OST0000 complete [ 4574.185275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4574.207876] Lustre: Skipped 1 previous similar message [ 4577.586898] Lustre: server umount lustre-MDT0000 complete [ 4579.305857] LustreError: 142526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.338866] LustreError: 142526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4581.707500] LustreError: 142508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588608 with bad export cookie 11351007485255358918 [ 4581.711828] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4581.717660] LustreError: 142508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4582.035106] Lustre: server umount lustre-MDT0001 complete [ 4591.835421] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 19:03:37 (1784588617) [ 4598.979845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4614.499121] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4619.616476] LustreError: 146688:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 4645.281658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.290436] Lustre: Skipped 2 previous similar messages [ 4652.408332] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4660.977996] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4668.966180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.932725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4675.017370] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4682.625129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.121198] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4695.043829] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4700.010919] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4704.304989] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4704.552875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4709.780340] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4709.998792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4713.965794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4713.965948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4713.968221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4713.968229] Lustre: Skipped 3 previous similar messages [ 4714.007725] Lustre: Skipped 2 previous similar messages [ 4718.318302] Lustre: server umount lustre-MDT0000 complete [ 4719.078578] LustreError: 147724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.103798] LustreError: 147724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4722.010714] LustreError: 148467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588749 with bad export cookie 11351007485255359940 [ 4722.017951] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4722.045361] LustreError: 148467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4722.544778] Lustre: server umount lustre-MDT0001 complete [ 4736.489210] Lustre: server umount lustre-OST0000 complete [ 4745.154923] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 19:06:11 (1784588771) [ 4752.402673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4768.096834] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4773.216483] LustreError: 150884:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 4805.832134] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4814.888725] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4822.668563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4823.402578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4828.576374] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4838.000866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.733781] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4850.751787] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4856.643372] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4863.736743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4864.087809] Lustre: lustre-OST0001: new disk, initializing [ 4864.096548] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4864.108600] Lustre: 153757:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4871.140522] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4872.780245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4872.793468] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4872.916269] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4881.322148] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4885.987400] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4886.279819] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4890.939879] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4891.218395] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4893.688409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.692116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4893.698915] Lustre: Skipped 1 previous similar message [ 4893.708131] Lustre: Skipped 3 previous similar messages [ 4899.132818] Lustre: server umount lustre-OST0000 complete [ 4903.915968] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4917.216598] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4917.436838] Lustre: server umount lustre-OST0001 complete [ 4927.792224] Lustre: server umount lustre-MDT0000 complete [ 4931.692942] LustreError: 150890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784588958 with bad export cookie 11351007485255360892 [ 4931.705683] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4931.715106] LustreError: 150890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4932.278839] Lustre: server umount lustre-MDT0001 complete [ 4959.704241] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 19:09:45 (1784588985) [ 4961.773644] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4963.475075] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4965.415525] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 19:09:51 (1784588991) [ 4972.100802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4972.288931] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4972.321369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4972.498555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4972.503343] Lustre: Skipped 6 previous similar messages [ 4977.549077] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 4986.704293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4986.805381] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4986.834977] Lustre: 157319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4991.911484] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5000.138522] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5006.500498] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5013.445347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5013.705708] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5020.496649] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5021.325825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5028.257714] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5033.781413] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5034.050351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5039.877820] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5040.116129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5046.753726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5046.760753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.780990] Lustre: Skipped 4 previous similar messages [ 5046.788710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5046.797293] Lustre: Skipped 10 previous similar messages [ 5048.607438] Lustre: server umount lustre-OST0000 complete [ 5071.884630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5073.332641] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5073.351556] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5080.144454] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5087.736393] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5087.759170] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5088.575543] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5094.805453] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5095.018807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5099.656905] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5099.915276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5105.849356] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92c1c302a800.ost_server_uuid 50 [ 5107.797904] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff92c1c302a800.ost_server_uuid in FULL state after 0 sec [ 5108.063394] LustreError: 157325:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.45@tcp) returned error from blocking AST (req@00000000307bb61c x1871276229795968 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000070b73c7c/0x9d86e12ae9722ad6 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.45@tcp remote: 0xf6a433b56a7883c expref: 5 pid: 157324 timeout: 5208 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5108.115225] LustreError: lustre-MDT0000: A client on nid 192.168.204.45@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5108.123333] LustreError: 157314:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.45@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000070b73c7c/0x9d86e12ae9722ad6 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.45@tcp remote: 0xf6a433b56a7883c expref: 6 pid: 157324 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5113.313662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5113.327447] LustreError: Skipped 1 previous similar message [ 5119.209254] Lustre: server umount lustre-OST0000 complete [ 5129.982487] Lustre: server umount lustre-MDT0000 complete [ 5133.793528] LustreError: 157330:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5133.810237] LustreError: 157330:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 5133.894245] LustreError: 157309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589160 with bad export cookie 11351007485255362075 [ 5133.910169] LustreError: 157309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5133.917065] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.537093] Lustre: server umount lustre-MDT0001 complete [ 5151.632328] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5155.183674] Key type lgssc unregistered [ 5155.573796] LNet: 163437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5155.580523] LNetError: 163437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5155.603095] LNet: Removed LNI 192.168.204.145@tcp [ 5156.727918] Key type .llcrypt unregistered [ 5156.731053] Key type ._llcrypt unregistered [ 5185.076206] Key type ._llcrypt registered [ 5185.078516] Key type .llcrypt registered [ 5185.237371] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5186.192238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5186.259214] alg: No test for adler32 (adler32-zlib) [ 5187.348618] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 5187.685696] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5189.368186] Key type lgssc registered [ 5190.732140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5200.549973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5200.561438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5202.001858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5207.609830] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5211.115215] LustreError: 164850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5211.133373] LustreError: 164850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5216.241830] LustreError: 164851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5218.674017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5219.155671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5223.998801] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5232.524456] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5239.070185] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5246.142516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5246.618050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5247.668030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5252.719685] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5260.347439] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5269.196083] Lustre: Failing over lustre-MDT0000 [ 5269.475810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5269.483537] LustreError: 166068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5269.568747] Lustre: server umount lustre-MDT0000 complete [ 5274.594071] LustreError: 165579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5274.613425] LustreError: 165579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5279.714499] LustreError: 164850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5279.730516] LustreError: 164850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5281.814929] LustreError: 164831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.45@tcp arrived at 1784589308 with bad export cookie 9536130482434669303 [ 5281.842421] LustreError: 164831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5288.946078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.102771] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5289.365230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5289.419184] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5294.010394] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5294.564961] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5294.571762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5295.980171] Lustre: lustre-MDT0000: Denying connection for new client 15a5544f-6561-4254-b74d-1b716b6c2cde (at 192.168.204.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5301.196442] Lustre: lustre-MDT0000: Denying connection for new client 15a5544f-6561-4254-b74d-1b716b6c2cde (at 192.168.204.45@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 5304.805278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5304.808394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5304.845624] Lustre: Skipped 3 previous similar messages [ 5304.865862] Lustre: Skipped 1 previous similar message [ 5309.927859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5309.942069] Lustre: Skipped 1 previous similar message [ 5310.851861] Lustre: server umount lustre-OST0000 complete [ 5315.145192] LustreError: 168581:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5315.150950] Lustre: 167930:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5315.155373] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5315.167230] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5315.174404] LustreError: 167930:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004b97627b x1871277419705728/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5315.183297] LustreError: 167930:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5315.189684] LustreError: 167930:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5315.196417] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5315.863228] Lustre: server umount lustre-MDT0000 complete [ 5320.161471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5320.171788] LustreError: 164846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5320.181495] LustreError: 164846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5320.253195] LustreError: 164159:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5320.256562] LustreError: 165587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589347 with bad export cookie 9536130482434669408 [ 5320.260195] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5320.292559] LustreError: 165587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5320.619144] Lustre: server umount lustre-MDT0001 complete [ 5332.818800] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5335.910884] Key type lgssc unregistered [ 5336.385542] LNet: 169334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5336.391796] LNetError: 169334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5336.419422] LNet: Removed LNI 192.168.204.145@tcp [ 5337.359910] Key type .llcrypt unregistered [ 5337.362934] Key type ._llcrypt unregistered [ 5361.168304] Key type ._llcrypt registered [ 5361.173637] Key type .llcrypt registered [ 5361.279919] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5362.486730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5362.679418] alg: No test for adler32 (adler32-zlib) [ 5363.697562] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 5363.929044] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5365.632339] Key type lgssc registered [ 5366.984644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5374.575316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5374.609172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5376.098751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5380.894501] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5386.724817] LustreError: 170747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.750276] LustreError: 170747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5388.434779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.736753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5392.230852] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5400.277913] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5406.458784] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5413.456556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5413.813724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5419.473797] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5422.038745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5428.250434] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5435.363848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5435.369689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5435.378892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5437.411660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5437.422749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5437.441171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.590828] Lustre: server umount lustre-OST0000 complete [ 5447.649731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5447.652524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5447.657114] Lustre: Skipped 1 previous similar message [ 5447.662064] Lustre: Skipped 1 previous similar message [ 5451.720260] Lustre: server umount lustre-MDT0000 complete [ 5452.769336] LustreError: 170743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.781239] LustreError: 170743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5455.431595] LustreError: 170728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589482 with bad export cookie 12936484855653512339 [ 5455.435676] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.445349] LustreError: 170728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5455.761700] Lustre: server umount lustre-MDT0001 complete [ 5467.326933] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5470.484198] Key type lgssc unregistered [ 5470.790731] LNet: 174279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5470.804377] LNetError: 174279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5470.835397] LNet: Removed LNI 192.168.204.145@tcp [ 5471.706376] Key type .llcrypt unregistered [ 5471.710910] Key type ._llcrypt unregistered [ 5503.108648] Key type ._llcrypt registered [ 5503.111827] Key type .llcrypt registered [ 5503.274644] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5504.670502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5504.741213] alg: No test for adler32 (adler32-zlib) [ 5505.912869] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 5506.195366] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5507.896286] Key type lgssc registered [ 5509.167124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5516.877599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5516.910605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.362784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5522.693140] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5528.545182] LustreError: 175981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5528.563592] LustreError: 175981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5531.465802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.904980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5536.585549] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5545.283811] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5551.094852] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5558.735559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5559.225714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.334659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5565.701654] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5576.245186] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5586.913647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.925242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.931484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5586.944631] Lustre: Skipped 1 previous similar message [ 5586.959413] Lustre: Skipped 1 previous similar message [ 5591.514331] Lustre: server umount lustre-OST0000 complete [ 5597.158149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.164824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5597.193976] Lustre: Skipped 1 previous similar message [ 5597.217376] Lustre: Skipped 1 previous similar message [ 5602.273397] LustreError: 175977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5602.301225] LustreError: 175977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5602.373685] Lustre: server umount lustre-MDT0000 complete [ 5606.585069] LustreError: 176715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589633 with bad export cookie 12373923905503362490 [ 5606.603943] LustreError: 176715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5606.615530] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5607.197129] Lustre: server umount lustre-MDT0001 complete [ 5619.006932] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5622.549148] Key type lgssc unregistered [ 5622.888776] LNet: 179516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5622.895562] LNetError: 179516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5622.930095] LNet: Removed LNI 192.168.204.145@tcp [ 5623.809468] Key type .llcrypt unregistered [ 5623.814583] Key type ._llcrypt unregistered [ 5648.341345] Key type ._llcrypt registered [ 5648.343819] Key type .llcrypt registered [ 5648.481887] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5649.639482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5649.817194] alg: No test for adler32 (adler32-zlib) [ 5650.845935] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 5651.127491] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5652.840201] Key type lgssc registered [ 5654.151767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5665.088578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5665.115991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5666.655594] Lustre: *** cfs_fail_loc=135, val=0*** [ 5666.757476] LustreError: 181040:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5666.770719] LustreError: 181040:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.145@tcp: cfg command failed: rc = -2 [ 5666.785400] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5666.785400] [ 5666.800692] LustreError: MGC192.168.204.145@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5666.819436] LustreError: 181001:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5666.826075] LustreError: 181001:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5666.834639] LustreError: 181001:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5666.841538] LustreError: 181001:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5666.916738] Lustre: server umount lustre-MDT0000 complete [ 5666.919527] LustreError: 181001:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5673.378699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.911890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.562234] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5683.173088] LustreError: 181413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5683.212193] LustreError: 181413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5687.116898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.580859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5692.144507] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5699.389935] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5704.511917] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5708.258200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5708.265669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5708.282383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5708.311218] Lustre: Skipped 1 previous similar message [ 5708.330044] Lustre: Skipped 1 previous similar message [ 5712.843080] Lustre: server umount lustre-MDT0000 complete [ 5713.380682] LustreError: 181412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5713.398588] LustreError: 181412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5714.422885] LustreError: 181412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.446050] LustreError: 181412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5716.910898] LustreError: 181394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589743 with bad export cookie 2392983624594666153 [ 5716.920926] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5717.293732] Lustre: server umount lustre-MDT0001 complete [ 5727.654785] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5730.644947] Key type lgssc unregistered [ 5730.922231] LNet: 183733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5730.934663] LNetError: 183733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5732.006777] LNet: Removed LNI 192.168.204.145@tcp [ 5732.986410] Key type .llcrypt unregistered [ 5732.992193] Key type ._llcrypt unregistered [ 5754.405922] Key type ._llcrypt registered [ 5754.407783] Key type .llcrypt registered [ 5754.512328] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5754.842559] Key type .llcrypt unregistered [ 5754.845866] Key type ._llcrypt unregistered [ 5772.764566] Key type ._llcrypt registered [ 5772.766280] Key type .llcrypt registered [ 5772.928464] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5774.158329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5774.202106] alg: No test for adler32 (adler32-zlib) [ 5775.365826] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 5775.617553] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5777.344403] Key type lgssc registered [ 5778.896280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5786.718923] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5786.763307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5802.400634] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5807.520644] LustreError: 185866:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.145@tcp: failed processing log, type 4: rc = -110 [ 5834.276459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5840.527544] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5847.299133] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5853.773188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.868587] LustreError: 185202:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000091532219 x1871278036224256/t0(0) o250->MGC192.168.204.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5854.207070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5854.258328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5858.692343] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5859.299025] LustreError: 185891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5859.320466] LustreError: 185891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5864.423541] LustreError: 185892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5866.848673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5867.280693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5871.458605] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5878.677264] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5883.680789] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5886.150522] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5899.240628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.250397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5899.258792] Lustre: Skipped 1 previous similar message [ 5904.354022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5904.363114] Lustre: Skipped 1 previous similar message [ 5904.484519] Lustre: server umount lustre-OST0000 complete [ 5908.449679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5908.455524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5908.464314] Lustre: Skipped 1 previous similar message [ 5908.468424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5909.481101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5914.559537] Lustre: server umount lustre-MDT0000 complete [ 5914.596154] LustreError: 186903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.609421] LustreError: 186903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5918.263469] LustreError: 185874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589945 with bad export cookie 3117909754876617556 [ 5918.272209] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5918.281812] LustreError: 185874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5918.537939] Lustre: server umount lustre-MDT0001 complete [ 5929.415708] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 5931.960246] Key type lgssc unregistered [ 5932.253248] LNet: 189880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5932.258891] LNetError: 189880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5932.275165] LNet: Removed LNI 192.168.204.145@tcp [ 5933.328502] Key type .llcrypt unregistered [ 5933.331305] Key type ._llcrypt unregistered [ 5956.697695] Key type ._llcrypt registered [ 5956.700838] Key type .llcrypt registered [ 5956.800240] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 5957.912142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5958.019419] alg: No test for adler32 (adler32-zlib) [ 5959.114345] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 5959.381146] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 5961.056217] Key type lgssc registered [ 5962.192733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5969.794782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5969.847856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5971.446560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5975.976476] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5981.671839] LustreError: 191291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5981.687392] LustreError: 191291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5984.058620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5984.335747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5988.136723] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 5994.563869] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6000.750380] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6007.787071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6008.305638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6014.646569] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6016.533905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6022.126234] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6026.385968] Lustre: Failing over lustre-MDT0000 [ 6026.721242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6026.727782] LustreError: 192363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.741856] Lustre: Skipped 2 previous similar messages [ 6026.770739] LustreError: 192363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6026.792537] Lustre: server umount lustre-MDT0000 complete [ 6028.788065] LustreError: 191285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6031.843714] LustreError: 192363:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6031.877379] LustreError: 192363:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6036.961696] LustreError: 191291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6036.981719] LustreError: 191291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6045.191398] LustreError: 191285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6045.208812] LustreError: 191285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6045.754603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6045.975967] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6046.154399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.201987] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6049.220860] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6050.249495] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6051.312722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6051.333182] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6051.370982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6054.263341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6062.348288] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6075.797333] Lustre: server umount lustre-OST0000 complete [ 6082.019892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6082.047792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.140315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.147023] Lustre: Skipped 2 previous similar messages [ 6089.321700] Lustre: server umount lustre-MDT0000 complete [ 6091.249727] Lustre: 190602:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784590102/real 1784590102] req@000000007179821f x1871278229216640/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784590118 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6091.314283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.355576] Lustre: Skipped 1 previous similar message [ 6091.381946] LustreError: 191291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.393407] Lustre: 190602:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784590103/real 1784590103] req@00000000ed91cbec x1871278229217152/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784590119 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6093.675516] LustreError: 191272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590120 with bad export cookie 7045781973021506604 [ 6093.684510] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6093.694717] LustreError: 191272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6093.995831] Lustre: server umount lustre-MDT0001 complete [ 6107.617698] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6111.656783] Key type lgssc unregistered [ 6112.219830] LNet: 196200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6112.227548] LNetError: 196200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6112.259964] LNet: Removed LNI 192.168.204.145@tcp [ 6113.357548] Key type .llcrypt unregistered [ 6113.359646] Key type ._llcrypt unregistered [ 6140.363183] Key type ._llcrypt registered [ 6140.370243] Key type .llcrypt registered [ 6140.506771] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6142.286257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6142.327415] alg: No test for adler32 (adler32-zlib) [ 6143.470889] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 6143.936124] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6145.768211] Key type lgssc registered [ 6147.195252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6156.335664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6156.353885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6157.914190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6163.363506] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6167.022192] LustreError: 197617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.047928] LustreError: 197617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6172.138993] LustreError: 197618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6172.305116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.754741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6177.880829] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6185.012896] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6191.137552] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6197.781788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6198.106273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6202.239122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6203.840151] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6210.862715] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6214.699355] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6217.915210] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6234.346933] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6243.306650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6243.317975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6243.330505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6243.344334] Lustre: Skipped 1 previous similar message [ 6243.359436] Lustre: Skipped 1 previous similar message [ 6246.173347] Lustre: server umount lustre-OST0000 complete [ 6253.539441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6253.551104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6253.565588] Lustre: Skipped 1 previous similar message [ 6253.577877] Lustre: Skipped 1 previous similar message [ 6255.902423] Lustre: server umount lustre-MDT0000 complete [ 6258.659114] LustreError: 197617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6258.682583] LustreError: 197617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6259.565822] LustreError: 200151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590286 with bad export cookie 17415702824676416652 [ 6259.572931] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6259.581804] LustreError: 200151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6259.898686] Lustre: server umount lustre-MDT0001 complete [ 6270.043640] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6272.579164] Key type lgssc unregistered [ 6272.832635] LNet: 201481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6272.838571] LNetError: 201481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6272.861079] LNet: Removed LNI 192.168.204.145@tcp [ 6273.700693] Key type .llcrypt unregistered [ 6273.704128] Key type ._llcrypt unregistered [ 6292.541277] Key type ._llcrypt registered [ 6292.547181] Key type .llcrypt registered [ 6292.709911] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6293.787211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6293.908538] alg: No test for adler32 (adler32-zlib) [ 6294.927721] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 6295.124594] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6296.816524] Key type lgssc registered [ 6298.018222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6305.948376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6305.968260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.451449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6312.689465] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6317.031122] LustreError: 202751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6317.058519] LustreError: 202751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6322.148671] LustreError: 202750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6322.212631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6322.461893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6328.686707] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6336.973343] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6343.032903] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6351.130744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6351.612414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6352.627356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6361.326582] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6369.893045] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6375.792306] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6389.950192] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6398.893753] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6405.082235] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6414.306146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.322403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.325746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6414.353168] Lustre: Skipped 1 previous similar message [ 6419.427018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6419.443357] Lustre: Skipped 1 previous similar message [ 6424.551762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6424.562749] Lustre: Skipped 1 previous similar message [ 6427.104638] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6427.292603] Lustre: server umount lustre-OST0000 complete [ 6434.790453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.799699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6434.818073] Lustre: Skipped 2 previous similar messages [ 6437.766942] Lustre: server umount lustre-MDT0000 complete [ 6439.908614] LustreError: 202750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.932167] LustreError: 202750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6442.141273] LustreError: 205875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590469 with bad export cookie 16998145018587674823 [ 6442.143049] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6442.152096] LustreError: 205875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6442.580425] Lustre: server umount lustre-MDT0001 complete [ 6453.820514] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6456.929976] Key type lgssc unregistered [ 6457.204649] LNet: 207630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6457.248584] LNetError: 207630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6457.269380] LNet: Removed LNI 192.168.204.145@tcp [ 6458.642705] Key type .llcrypt unregistered [ 6458.645947] Key type ._llcrypt unregistered [ 6485.521301] Key type ._llcrypt registered [ 6485.523947] Key type .llcrypt registered [ 6485.631363] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6486.336684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6486.424310] alg: No test for adler32 (adler32-zlib) [ 6487.749848] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 6488.096242] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6489.856296] Key type lgssc registered [ 6491.410982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6499.949639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6499.963648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6501.504155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.220777] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6511.090137] LustreError: 209044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6511.110451] LustreError: 209044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6513.613994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.034346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6518.872725] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6525.978441] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6531.524113] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6538.022686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6538.452457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6544.556170] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6546.691702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6551.889199] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6567.400108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6567.401469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6567.413745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6567.427752] Lustre: Skipped 1 previous similar message [ 6571.938944] Lustre: server umount lustre-OST0000 complete [ 6577.637867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6577.655883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6577.660075] Lustre: Skipped 1 previous similar message [ 6582.158632] Lustre: server umount lustre-MDT0000 complete [ 6582.753386] LustreError: 209772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6582.777411] LustreError: 209772:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6585.880794] LustreError: 209026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590612 with bad export cookie 16180089650131485478 [ 6585.888843] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6585.897997] LustreError: 209026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6586.189917] Lustre: server umount lustre-MDT0001 complete [ 6596.143928] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6599.106509] Key type lgssc unregistered [ 6599.396551] LNet: 212727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6599.407839] LNetError: 212727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6599.423065] LNet: Removed LNI 192.168.204.145@tcp [ 6600.482702] Key type .llcrypt unregistered [ 6600.487910] Key type ._llcrypt unregistered [ 6620.203267] Key type ._llcrypt registered [ 6620.205503] Key type .llcrypt registered [ 6620.314835] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6621.382636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6621.440063] alg: No test for adler32 (adler32-zlib) [ 6622.544683] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 6622.768392] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6624.465266] Key type lgssc registered [ 6625.657302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6633.551194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6633.583432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.019483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.823803] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6644.196127] LustreError: 213996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.234171] LustreError: 213996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6649.180789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6649.319921] LustreError: 213997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6649.715948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6654.751164] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6663.122143] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6669.375890] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6676.333804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6676.674806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6679.745476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6682.637814] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6690.217498] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6704.613280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.627177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6704.636908] Lustre: Skipped 1 previous similar message [ 6704.669384] Lustre: Skipped 1 previous similar message [ 6705.732220] Lustre: server umount lustre-OST0000 complete [ 6709.731063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6709.736603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6709.743897] Lustre: Skipped 1 previous similar message [ 6709.759474] Lustre: Skipped 1 previous similar message [ 6714.853841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6714.871689] Lustre: Skipped 1 previous similar message [ 6717.569867] Lustre: server umount lustre-MDT0000 complete [ 6719.970159] LustreError: 213993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6719.992840] LustreError: 213993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6721.298402] LustreError: 213977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590748 with bad export cookie 18337108383573563651 [ 6721.309289] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6721.573873] Lustre: server umount lustre-MDT0001 complete [ 6733.572328] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6736.401779] Key type lgssc unregistered [ 6736.776814] LNet: 217679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6736.787153] LNetError: 217679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6736.814059] LNet: Removed LNI 192.168.204.145@tcp [ 6737.715589] Key type .llcrypt unregistered [ 6737.720502] Key type ._llcrypt unregistered [ 6757.076273] Key type ._llcrypt registered [ 6757.081163] Key type .llcrypt registered [ 6757.236500] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6758.543441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6758.572220] alg: No test for adler32 (adler32-zlib) [ 6759.760204] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 6760.029065] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6761.768182] Key type lgssc registered [ 6762.837571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6770.951348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6770.970695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.393574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.322325] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6782.959267] LustreError: 218945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6782.988498] LustreError: 218945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6785.993465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6786.535027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6791.053917] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6797.966792] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6804.969900] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6811.714680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6812.189356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.334977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6818.572260] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6825.757770] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6834.269078] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6834.395767] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6834.735403] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6839.741886] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6845.996425] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6847.393395] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6858.911972] Lustre: Permanently deactivating lustre-OST0001 [ 6858.935780] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6868.024329] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6874.341411] Lustre: Permanently reactivating lustre-OST0001 [ 6874.361896] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6874.368649] Lustre: Skipped 2 previous similar messages [ 6875.680774] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6875.695352] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6875.704359] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6875.731133] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6875.831251] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6884.724596] Lustre: server umount lustre-OST0001 complete [ 6886.373823] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6886.374234] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.386262] LustreError: Skipped 1 previous similar message [ 6886.390505] LustreError: 221296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.415820] Lustre: Skipped 2 previous similar messages [ 6886.446505] LustreError: 221296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6887.398335] LustreError: 220769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6888.934141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6888.947613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6888.969151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6892.514565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6892.520171] LustreError: 220768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.523066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6892.548381] LustreError: 220768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6894.564846] Lustre: server umount lustre-OST0000 complete [ 6899.175650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6899.184056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.203097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6899.206152] Lustre: Skipped 1 previous similar message [ 6902.754255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6902.765398] Lustre: Skipped 1 previous similar message [ 6904.002649] Lustre: server umount lustre-MDT0000 complete [ 6907.395689] LustreError: 220792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590934 with bad export cookie 10985405677158922673 [ 6907.410122] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6907.413621] LustreError: 220792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6907.805993] Lustre: server umount lustre-MDT0001 complete [ 6918.186583] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 6920.908496] Key type lgssc unregistered [ 6921.141506] LNet: 224003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6921.146971] LNetError: 224003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6921.159798] LNet: Removed LNI 192.168.204.145@tcp [ 6921.979460] Key type .llcrypt unregistered [ 6921.981460] Key type ._llcrypt unregistered [ 6957.681894] Key type ._llcrypt registered [ 6957.685212] Key type .llcrypt registered [ 6957.788857] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 6958.722558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6958.837116] alg: No test for adler32 (adler32-zlib) [ 6959.861145] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 6960.146326] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 6961.832316] Key type lgssc registered [ 6962.839805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6968.965858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6969.011890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.265630] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6970.284212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6970.477485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6974.677723] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6981.647620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6981.735094] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6981.773145] Lustre: 226202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6982.029639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6985.661162] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 6991.838590] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6996.804457] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7002.215754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7002.391595] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7002.637156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7007.517221] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7009.400674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7015.046391] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7018.882175] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7027.578933] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7036.216748] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7045.967718] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7063.498493] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7063.504212] Lustre: Skipped 1 previous similar message [ 7084.159927] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7084.166218] Lustre: Skipped 2 previous similar messages [ 7116.470283] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7116.477846] Lustre: Skipped 3 previous similar messages [ 7184.126034] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7184.142627] Lustre: Skipped 7 previous similar messages [ 7202.277487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7202.282843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7202.296541] Lustre: Skipped 1 previous similar message [ 7202.313662] Lustre: Skipped 1 previous similar message [ 7206.384989] Lustre: server umount lustre-OST0000 complete [ 7212.513207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7212.514930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7212.531460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7212.557945] Lustre: Skipped 1 previous similar message [ 7215.782848] Lustre: server umount lustre-MDT0000 complete [ 7218.888119] LustreError: 228062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591245 with bad export cookie 13413091349155083618 [ 7218.893689] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7218.898068] LustreError: 228062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7219.313863] Lustre: server umount lustre-MDT0001 complete [ 7229.333443] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7231.906405] Key type lgssc unregistered [ 7232.197667] LNet: 231018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7232.203317] LNetError: 231018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7232.217937] LNet: Removed LNI 192.168.204.145@tcp [ 7232.961460] Key type .llcrypt unregistered [ 7232.962925] Key type ._llcrypt unregistered [ 7253.163830] Key type ._llcrypt registered [ 7253.167635] Key type .llcrypt registered [ 7253.273577] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7254.452594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7254.471812] alg: No test for adler32 (adler32-zlib) [ 7255.635130] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 7255.911678] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7257.624245] Key type lgssc registered [ 7258.624687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7265.503924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7265.521671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7266.905795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7271.189359] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7276.013488] LustreError: 232411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7276.040058] LustreError: 232411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7279.487312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.882124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7284.256831] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7290.792556] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7296.374814] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7302.316513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7302.696495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7308.032909] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_default_debug -1 all [ 7311.929275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7314.477898] Lustre: DEBUG MARKER: oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7329.547212] Lustre: server umount lustre-OST0000 complete [ 7336.419612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7336.428126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.437916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7337.442828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7337.444314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7337.469392] Lustre: Skipped 1 previous similar message [ 7342.561724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7342.569920] Lustre: Skipped 1 previous similar message [ 7346.656149] Lustre: 231722:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784591357/real 1784591357] req@0000000068ffbc46 x1871279588149504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1784591373 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7346.683938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7346.698967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7346.705247] Lustre: Skipped 1 previous similar message [ 7347.168132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7347.292910] Lustre: server umount lustre-MDT0000 complete [ 7348.706416] Lustre: 231723:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784591359/real 1784591359] req@0000000083eb3d05 x1871279588150656/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784591375 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7350.202921] LustreError: 232393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591377 with bad export cookie 1692854351570448461 [ 7350.207572] LustreError: MGC192.168.204.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7350.218584] LustreError: 232393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7350.458540] Lustre: server umount lustre-MDT0001 complete [ 7359.461527] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7362.092635] Key type lgssc unregistered [ 7362.439267] LNet: 236099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7362.451819] LNetError: 236099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7362.470741] LNet: Removed LNI 192.168.204.145@tcp [ 7363.328888] Key type .llcrypt unregistered [ 7363.331446] Key type ._llcrypt unregistered [ 7383.590178] Key type ._llcrypt registered [ 7383.592199] Key type .llcrypt registered [ 7383.683608] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7384.684318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7384.810904] alg: No test for adler32 (adler32-zlib) [ 7385.890403] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 7386.160669] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7387.824158] Key type lgssc registered [ 7388.808113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7656.656704] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7658.673922] Key type lgssc unregistered [ 7658.919204] LNet: 238115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7658.932215] LNetError: 238115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7658.958814] LNet: Removed LNI 192.168.204.145@tcp [ 7659.576538] Key type .llcrypt unregistered [ 7659.578637] Key type ._llcrypt unregistered [ 7683.150643] Key type ._llcrypt registered [ 7683.153087] Key type .llcrypt registered [ 7683.275590] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing set_hostid [ 7697.850046] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing load_modules_local [ 7698.687525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7698.739779] alg: No test for adler32 (adler32-zlib) [ 7699.748979] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 7699.936141] LNet: Added LNI 192.168.204.145@tcp [8/256/0/180] [ 7701.608216] Key type lgssc registered [ 7702.530346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7725.608829] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7405 sec ========== 19:55:52 (1784591752) [ 7726.976885] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:55:53 (1784591753) === [ 7729.797437] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:55:56 (1784591756) === [ 7758.424688] Lustre: DEBUG MARKER: oleg445-server.virtnet: executing unload_modules_local [ 7760.500215] Key type lgssc unregistered [ 7760.711733] LNet: 243350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7760.720792] LNetError: 243350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7760.736555] LNet: Removed LNI 192.168.204.145@tcp [ 7761.284888] Key type .llcrypt unregistered [ 7761.287453] Key type ._llcrypt unregistered