[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478811658 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002358] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006999] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009127] LSM: Security Framework initializing [ 0.010047] Yama: becoming mindful. [ 0.011036] SELinux: Initializing. [ 0.012075] *** VALIDATE selinux *** [ 0.020769] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024585] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026057] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028102] *** VALIDATE tmpfs *** [ 0.030298] *** VALIDATE proc *** [ 0.031223] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041362] debug: unmapping init [mem 0xffffffff84659000-0xffffffff84660fff] [ 0.043193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044634] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051013] ... event mask: 000000070000000f [ 0.052297] rcu: Hierarchical SRCU implementation. [ 0.054397] smp: Bringing up secondary CPUs ... [ 0.055516] x86: Booting SMP configuration: [ 0.056023] .... node #0, CPUs: #1 #2 #3 [ 0.059250] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142913] node 0 deferred pages initialised in 78ms [ 0.145578] devtmpfs: initialized [ 0.146193] x86/mm: Memory block size: 128MB [ 0.148666] gcov: version magic: 0x41383552 [ 0.149615] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153295] pinctrl core: initialized pinctrl subsystem [ 0.154133] [ 0.154544] ************************************************************* [ 0.156011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158010] ** ** [ 0.161012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164013] ** ** [ 0.167011] ** This means that this kernel is built to expose internal ** [ 0.169012] ** IOMMU data structures, which may compromise security on ** [ 0.172014] ** your system. ** [ 0.175011] ** ** [ 0.178017] ** If you see this message and you are not debugging the ** [ 0.180012] ** kernel, report this immediately to your vendor! ** [ 0.182012] ** ** [ 0.185014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187012] ************************************************************* [ 0.190736] NET: Registered protocol family 16 [ 0.192410] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201029] cpuidle: using governor menu [ 0.203508] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206549] PCI: Using configuration type 1 for base access [ 0.208135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218079] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221108] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225078] cryptd: max_cpu_qlen set to 1000 [ 0.229239] ACPI: Added _OSI(Module Device) [ 0.230012] ACPI: Added _OSI(Processor Device) [ 0.232016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239341] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246223] ACPI: Interpreter enabled [ 0.247053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248011] ACPI: Using IOAPIC for interrupt routing [ 0.250089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252478] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.262433] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.265044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.271074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278279] acpiphp: Slot [2] registered [ 0.280141] acpiphp: Slot [5] registered [ 0.281118] acpiphp: Slot [6] registered [ 0.283104] acpiphp: Slot [7] registered [ 0.285130] acpiphp: Slot [8] registered [ 0.286119] acpiphp: Slot [9] registered [ 0.287119] acpiphp: Slot [10] registered [ 0.289133] acpiphp: Slot [3] registered [ 0.291131] acpiphp: Slot [4] registered [ 0.292117] acpiphp: Slot [11] registered [ 0.294131] acpiphp: Slot [12] registered [ 0.295093] acpiphp: Slot [13] registered [ 0.297194] acpiphp: Slot [14] registered [ 0.298100] acpiphp: Slot [15] registered [ 0.300126] acpiphp: Slot [16] registered [ 0.301109] acpiphp: Slot [17] registered [ 0.303093] acpiphp: Slot [18] registered [ 0.305093] acpiphp: Slot [19] registered [ 0.306101] acpiphp: Slot [20] registered [ 0.307097] acpiphp: Slot [21] registered [ 0.309109] acpiphp: Slot [22] registered [ 0.310185] acpiphp: Slot [23] registered [ 0.312087] acpiphp: Slot [24] registered [ 0.314105] acpiphp: Slot [25] registered [ 0.315079] acpiphp: Slot [26] registered [ 0.316095] acpiphp: Slot [27] registered [ 0.318114] acpiphp: Slot [28] registered [ 0.320194] acpiphp: Slot [29] registered [ 0.321097] acpiphp: Slot [30] registered [ 0.322067] acpiphp: Slot [31] registered [ 0.324060] PCI host bridge to bus 0000:00 [ 0.325015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.333026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.336024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344099] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348418] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359010] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.364562] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.367016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373985] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376674] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.381724] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.398965] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403975] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409425] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.417028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425027] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.455709] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.489017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.498715] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.509024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.516022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.540081] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.551168] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.558012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.568017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.586015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.594827] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.601016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.608017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.625015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.635140] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.644030] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.650022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.669023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.682989] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.685468] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.688387] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.693392] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.697310] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.702682] iommu: Default domain type: Passthrough [ 0.706198] SCSI subsystem initialized [ 0.708223] ACPI: bus type USB registered [ 0.710198] usbcore: registered new interface driver usbfs [ 0.713120] usbcore: registered new interface driver hub [ 0.715119] usbcore: registered new device driver usb [ 0.718244] pps_core: LinuxPPS API ver. 1 registered [ 0.720016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.724093] PTP clock support registered [ 0.726188] EDAC MC: Ver: 3.0.0 [ 0.728159] PCI: Using ACPI for IRQ routing [ 0.729772] NetLabel: Initializing [ 0.731013] NetLabel: domain hash size = 128 [ 0.732010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.734081] NetLabel: unlabeled traffic allowed by default [ 0.736150] vgaarb: loaded [ 0.738403] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.740027] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.753605] clocksource: Switched to clocksource kvm-clock [ 0.859155] VFS: Disk quotas dquot_6.6.0 [ 0.860609] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.862981] *** VALIDATE ramfs *** [ 0.864261] *** VALIDATE hugetlbfs *** [ 0.866451] pnp: PnP ACPI init [ 0.868770] pnp: PnP ACPI: found 6 devices [ 0.889375] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.892913] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.895021] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.897248] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.899650] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.902087] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.905234] NET: Registered protocol family 2 [ 0.907745] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.913186] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.917495] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.922924] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.927315] TCP: Hash tables configured (established 65536 bind 65536) [ 0.931107] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.934951] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.938349] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941610] NET: Registered protocol family 1 [ 0.944717] RPC: Registered named UNIX socket transport module. [ 0.946964] RPC: Registered udp transport module. [ 0.949075] RPC: Registered tcp transport module. [ 0.950815] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.953501] NET: Registered protocol family 44 [ 0.955386] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.957739] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.959868] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.962280] PCI: CLS 0 bytes, default 64 [ 0.963863] Unpacking initramfs... [ 2.367897] debug: unmapping init [mem 0xffff89b73cc54000-0xffff89b73ffbffff] [ 2.372205] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.373821] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.376138] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.862884] Initialise system trusted keyrings [ 2.864365] Key type blacklist registered [ 2.866333] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.875402] zbud: loaded [ 2.878114] *** VALIDATE nfs *** [ 2.878900] *** VALIDATE nfs4 *** [ 2.880279] pstore: using deflate compression [ 2.882878] Platform Keyring initialized [ 2.976899] NET: Registered protocol family 38 [ 2.978762] Key type asymmetric registered [ 2.980668] Asymmetric key parser 'x509' registered [ 2.982632] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.985093] io scheduler mq-deadline registered [ 2.986637] io scheduler kyber registered [ 2.987994] io scheduler bfq registered [ 2.989784] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.992395] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.995281] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.997733] ACPI: Power Button [PWRF] [ 3.002396] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.008450] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.020875] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.027902] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.040263] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.069648] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.096972] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.101691] Non-volatile memory driver v1.3 [ 3.103588] Linux agpgart interface v0.103 [ 3.130816] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.133560] vda: detected capacity change from 0 to 74702848 [ 3.145740] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.148023] vdb: detected capacity change from 0 to 1073741824 [ 3.160758] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.164034] vdc: detected capacity change from 0 to 2621440000 [ 3.177080] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.180141] vdd: detected capacity change from 0 to 2621440000 [ 3.193327] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.195553] vde: detected capacity change from 0 to 4294967296 [ 3.208277] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.210864] vdf: detected capacity change from 0 to 4294967296 [ 3.216764] libphy: Fixed MDIO Bus: probed [ 3.225564] usbcore: registered new interface driver usbserial_generic [ 3.227667] usbserial: USB Serial support registered for generic [ 3.230296] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.234511] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.236346] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.238787] mousedev: PS/2 mouse device common for all mice [ 3.241811] rtc_cmos 00:05: RTC can wake from S4 [ 3.244958] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.250084] rtc_cmos 00:05: registered as rtc0 [ 3.250264] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.251827] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.256000] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.258422] intel_pstate: CPU model not supported [ 3.263648] hid: raw HID events driver (C) Jiri Kosina [ 3.265715] usbcore: registered new interface driver usbhid [ 3.267470] usbhid: USB HID core driver [ 3.269180] drop_monitor: Initializing network drop monitor service [ 3.270851] Initializing XFRM netlink socket [ 3.273422] NET: Registered protocol family 10 [ 3.277046] Segment Routing with IPv6 [ 3.278350] NET: Registered protocol family 17 [ 3.281755] mpls_gso: MPLS GSO support [ 3.286738] RAS: Correctable Errors collector initialized. [ 3.289082] AVX version of gcm_enc/dec engaged. [ 3.290899] AES CTR mode by8 optimization enabled [ 3.352891] sched_clock: Marking stable (3352838040, 0)->(4213772733, -860934693) [ 3.356426] registered taskstats version 1 [ 3.358429] Loading compiled-in X.509 certificates [ 3.360237] zswap: loaded using pool lzo/zbud [ 3.379883] Key type big_key registered [ 3.391466] Key type encrypted registered [ 3.393616] ima: No TPM chip found, activating TPM-bypass! [ 3.397120] ima: Allocated hash algorithm: sha1 [ 3.399275] ima: No architecture policies found [ 3.401291] evm: Initialising EVM extended attributes: [ 3.404147] evm: security.selinux [ 3.406425] evm: security.ima [ 3.408114] evm: security.capability [ 3.409932] evm: HMAC attrs: 0x1 [ 3.412702] rtc_cmos 00:05: setting system clock to 2026-08-13 13:17:52 UTC (1786627072) [ 3.420341] debug: unmapping init [mem 0xffffffff85603000-0xffffffff857fffff] [ 3.423468] debug: unmapping init [mem 0xffffffff84382000-0xffffffff84658fff] [ 3.433317] Write protecting the kernel read-only data: 28672k [ 3.438290] debug: unmapping init [mem 0xffffffff82a03000-0xffffffff82bfffff] [ 3.441070] debug: unmapping init [mem 0xffffffff83314000-0xffffffff833fffff] [ 3.473881] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.479204] systemd[1]: Detected virtualization kvm. [ 3.480257] systemd[1]: Detected architecture x86-64. [ 3.481722] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.506905] systemd[1]: No hostname configured. [ 3.508645] systemd[1]: Set hostname to . [ 3.510863] random: systemd: uninitialized urandom read (16 bytes read) [ 3.513527] systemd[1]: Initializing machine ID from random generator. [ 3.643407] random: systemd: uninitialized urandom read (16 bytes read) [ 3.646508] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.651217] random: systemd: uninitialized urandom read (16 bytes read) [ 3.653970] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.658198] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.217372] device-mapper: uevent: version 1.0.3 [ 4.218876] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.848886] virtio_net virtio0 ens2: renamed from eth0 [ 4.864972] random: fast init done [ 4.917071] scsi host0: ata_piix [ 4.925105] scsi host1: ata_piix [ 4.926952] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.930569] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.132050] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.915692] random: crng init done [ 9.917123] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.194978] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.347257] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.617684] SELinux: Disabled at runtime. [ 11.676907] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.687241] systemd[1]: Detected virtualization kvm. [ 11.690115] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.189541] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.193688] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.197772] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.202367] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.206318] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.212665] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.218650] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped targ[ 12.361789] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.667485] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.969319] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.990667] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.091377] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.105340] EDAC sbridge: Ver: 1.1.2 [ 14.871962] Key type dns_resolver registered [ 15.161783] NFS: Registering the id_resolver key type [ 15.163923] Key type id_resolver registered [ 15.165769] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg357-server login: [ 41.472926] libcfs: loading out-of-tree module taints kernel. [ 41.492259] Key type ._llcrypt registered [ 41.494696] Key type .llcrypt registered [ 41.537438] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 50.676490] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 51.543773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.555219] alg: No test for adler32 (adler32-zlib) [ 53.027769] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 54.047554] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 55.855204] Key type lgssc registered [ 57.567107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.723301] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 91.742011] hrtimer: interrupt took 2015566 ns [ 117.141437] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 132.096644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 132.158141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.468186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 133.501028] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 133.559188] Lustre: lustre-MDT0000: new disk, initializing [ 133.662222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 133.675328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.013312] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.939504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.039656] Lustre: 6491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 153.081191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 153.084304] Lustre: Skipped 1 previous similar message [ 153.185186] Lustre: lustre-MDT0001: new disk, initializing [ 153.241325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.268274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 153.273224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.377664] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.665934] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 173.686327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 173.967492] Lustre: lustre-OST0000: new disk, initializing [ 173.976449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 173.986792] Lustre: 8429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 174.077993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.880746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 177.890919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 177.957791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 180.181651] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.907491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.043709] Lustre: lustre-OST0001: new disk, initializing [ 194.051256] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.060520] Lustre: 9500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.131243] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.380379] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.728836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 200.736484] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 200.799933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.469529] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 215.982559] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 221.319951] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing check_logdir /tmp/testlogs/ [ 225.995454] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing yml_node [ 229.742763] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 232.061843] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 234.668787] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 236.398539] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 13 09:21:43 EDT 2026 [ 254.886701] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 256.713528] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 272.356834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.358037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 272.377596] Lustre: Skipped 3 previous similar messages [ 272.379769] Lustre: Skipped 2 previous similar messages [ 277.474354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 277.489787] Lustre: Skipped 4 previous similar messages [ 278.571463] Lustre: server umount lustre-MDT0000 complete [ 282.953101] LustreError: 6481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627352 with bad export cookie 732742885447893179 [ 282.957614] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 282.975286] LustreError: 6481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 287.719349] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.721023] LustreError: 11981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.721566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 287.776508] Lustre: Skipped 1 previous similar message [ 287.804709] LustreError: 11981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 289.544177] Lustre: server umount lustre-MDT0001 complete [ 294.664257] Lustre: server umount lustre-OST0000 complete [ 298.911195] Lustre: server umount lustre-OST0001 complete [ 306.739451] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 317.182152] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 355.591045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 355.849744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 355.897285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 355.977610] Lustre: lustre-MDT0000: new disk, initializing [ 356.081398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 356.109878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 360.317979] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 370.677376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 370.855168] Lustre: 15050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 370.920268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 370.924828] Lustre: Skipped 1 previous similar message [ 371.012766] Lustre: lustre-MDT0001: new disk, initializing [ 371.098603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 371.147764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 371.158034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 374.708645] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 384.834790] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 391.640477] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 400.603353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 400.904425] Lustre: lustre-OST0000: new disk, initializing [ 400.910142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 400.921126] Lustre: 17021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 401.022566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.199863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 402.211450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 402.308764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 407.429346] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 416.910080] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 421.361938] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 421.524860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 426.350668] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 426.596209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 433.125639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 433.131032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.134506] LustreError: Skipped 1 previous similar message [ 433.145237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 433.156833] Lustre: Skipped 1 previous similar message [ 433.184518] Lustre: Skipped 2 previous similar messages [ 434.683991] Lustre: server umount lustre-OST0000 complete [ 438.240362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.258954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 444.250423] Lustre: server umount lustre-MDT0000 complete [ 448.137155] LustreError: 15041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627517 with bad export cookie 732742885447894712 [ 448.140982] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 448.153977] LustreError: 15041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 448.523767] Lustre: server umount lustre-MDT0001 complete [ 451.826362] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:25:19 (1786627519) [ 457.998992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 458.592195] LustreError: 19284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 458.698980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.755379] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 463.841269] LustreError: 19285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 467.946399] LustreError: 19284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.620254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 477.686230] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 485.940915] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 492.864710] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 500.243045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 506.615325] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 516.286476] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 528.361210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.365576] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.399815] Lustre: Skipped 1 previous similar message [ 528.417803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.421360] Lustre: Skipped 4 previous similar messages [ 533.133859] Lustre: server umount lustre-OST0000 complete [ 538.602001] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.630224] Lustre: Skipped 2 previous similar messages [ 543.090256] Lustre: server umount lustre-MDT0000 complete [ 543.713859] LustreError: 19285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.734913] LustreError: 19285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 546.460192] LustreError: 19266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627615 with bad export cookie 732742885447895594 [ 546.469044] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 546.469961] LustreError: 19266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 546.769724] Lustre: server umount lustre-MDT0001 complete [ 557.964822] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 560.657143] Key type lgssc unregistered [ 560.905070] LNet: 22820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.911985] LNetError: 22820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.948231] LNet: Removed LNI 192.168.203.157@tcp [ 562.105142] Key type .llcrypt unregistered [ 562.107757] Key type ._llcrypt unregistered [ 596.565206] Key type ._llcrypt registered [ 596.571111] Key type .llcrypt registered [ 596.665199] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 597.718296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 597.833115] alg: No test for adler32 (adler32-zlib) [ 598.888665] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 599.135967] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 600.839699] Key type lgssc registered [ 601.819728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 608.858032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 608.883333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.179419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 614.280937] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 620.515775] LustreError: 24237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.540700] LustreError: 24237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 622.921332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.416044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 628.277766] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 636.848893] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 642.613142] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 649.447569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 649.874893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 655.059319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 656.023068] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 663.751847] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 678.888475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 678.892303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 678.893896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 678.902243] Lustre: Skipped 1 previous similar message [ 678.918304] Lustre: Skipped 1 previous similar message [ 680.949962] Lustre: server umount lustre-OST0000 complete [ 685.024839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 685.039298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.073502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 689.122369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 689.128599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.064799] Lustre: server umount lustre-MDT0000 complete [ 694.240787] LustreError: 24236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 694.275361] LustreError: 24236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 695.073199] LustreError: 24217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627764 with bad export cookie 2353716228507125437 [ 695.083066] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 695.087533] LustreError: 24217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 695.569782] Lustre: server umount lustre-MDT0001 complete [ 708.797360] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 711.978414] Key type lgssc unregistered [ 712.237917] LNet: 28215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.254470] LNetError: 28215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.282697] LNet: Removed LNI 192.168.203.157@tcp [ 713.206799] Key type .llcrypt unregistered [ 713.209893] Key type ._llcrypt unregistered [ 738.738780] Key type ._llcrypt registered [ 738.741540] Key type .llcrypt registered [ 738.838500] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 740.240407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 740.288153] alg: No test for adler32 (adler32-zlib) [ 741.317566] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 741.551772] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 743.199392] Key type lgssc registered [ 744.256773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 752.144210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 752.175136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.747185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 758.066400] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 763.874025] LustreError: 29629:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 763.900096] LustreError: 29629:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 767.049312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 767.288692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 771.702993] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 779.462584] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 785.692842] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 798.593250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 798.842691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 804.432814] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 811.907221] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 823.404794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 838.122180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 838.136511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 838.151390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 839.136058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 839.148664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 839.158033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.181249] Lustre: Skipped 1 previous similar message [ 841.428063] Lustre: server umount lustre-OST0000 complete [ 849.377158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 849.378928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.412276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 849.426993] Lustre: Skipped 1 previous similar message [ 854.499646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 854.510289] Lustre: Skipped 1 previous similar message [ 859.616419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 859.626634] Lustre: Skipped 1 previous similar message [ 860.127146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 860.343784] Lustre: server umount lustre-MDT0000 complete [ 864.554608] LustreError: 29609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786627933 with bad export cookie 4788020694506260188 [ 864.561584] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 864.566349] LustreError: 29609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 864.878848] Lustre: server umount lustre-MDT0001 complete [ 875.891477] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 878.641259] Key type lgssc unregistered [ 878.947505] LNet: 33535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.956720] LNetError: 33535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.982680] LNet: Removed LNI 192.168.203.157@tcp [ 879.982253] Key type .llcrypt unregistered [ 879.991192] Key type ._llcrypt unregistered [ 903.195186] Key type ._llcrypt registered [ 903.198382] Key type .llcrypt registered [ 903.313758] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 904.448227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 904.579723] alg: No test for adler32 (adler32-zlib) [ 905.641966] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 905.937020] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 907.616849] Key type lgssc registered [ 908.587649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 916.102630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 916.141938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.771657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.768841] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 927.203437] LustreError: 34951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.220555] LustreError: 34951:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 930.746933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.022438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 935.097560] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 943.143206] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 949.460703] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 956.832806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 957.230193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.347828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 964.241161] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 972.411837] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 986.086847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 986.095911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.109024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 987.110315] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 987.118480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.122864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 987.159438] Lustre: Skipped 1 previous similar message [ 992.231050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 992.240593] Lustre: Skipped 1 previous similar message [ 997.354305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 998.879157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 999.031154] Lustre: server umount lustre-OST0000 complete [ 1007.588749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.629784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1007.632489] Lustre: Skipped 1 previous similar message [ 1010.165210] Lustre: server umount lustre-MDT0000 complete [ 1012.708264] LustreError: 34950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.736689] LustreError: 34950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1015.503306] LustreError: 37447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628084 with bad export cookie 16897014055709319955 [ 1015.508825] LustreError: 37447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1015.508878] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.365969] Lustre: server umount lustre-MDT0001 complete [ 1028.393336] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1031.135273] Key type lgssc unregistered [ 1031.476337] LNet: 38513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.490381] LNetError: 38513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.510092] LNet: Removed LNI 192.168.203.157@tcp [ 1032.571662] Key type .llcrypt unregistered [ 1032.575231] Key type ._llcrypt unregistered [ 1058.431906] Key type ._llcrypt registered [ 1058.434528] Key type .llcrypt registered [ 1058.527893] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1059.283554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.338261] alg: No test for adler32 (adler32-zlib) [ 1060.492526] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1060.752764] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1062.440136] Key type lgssc registered [ 1063.598789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1071.856810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1071.894119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.355086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.033761] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1083.876035] LustreError: 39927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1083.898103] LustreError: 39927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1087.747804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.037289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1092.331701] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1100.433733] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1105.811761] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1112.237387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1112.552968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1118.729512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1119.013407] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1126.332554] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1134.052212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1134.053621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.060669] LustreError: Skipped 1 previous similar message [ 1134.062113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1134.070644] Lustre: Skipped 1 previous similar message [ 1138.963237] Lustre: server umount lustre-OST0000 complete [ 1159.649946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.669707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.676120] Lustre: Skipped 3 previous similar messages [ 1164.046714] Lustre: server umount lustre-MDT0000 complete [ 1164.773569] LustreError: 39928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.789241] LustreError: 39928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1167.407800] LustreError: 39908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628236 with bad export cookie 7896997064182105609 [ 1167.414992] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1167.422852] LustreError: 39908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1173.739474] Lustre: server umount lustre-MDT0001 complete [ 1184.498215] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1186.842631] Key type lgssc unregistered [ 1187.112698] LNet: 43639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1187.121565] LNetError: 43639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1187.143152] LNet: Removed LNI 192.168.203.157@tcp [ 1187.807669] Key type .llcrypt unregistered [ 1187.809583] Key type ._llcrypt unregistered [ 1209.237665] Key type ._llcrypt registered [ 1209.241150] Key type .llcrypt registered [ 1209.410931] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1210.200100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1210.237552] alg: No test for adler32 (adler32-zlib) [ 1211.277767] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1211.459772] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1213.087231] Key type lgssc registered [ 1214.168357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1221.892975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1221.921459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.651405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.686929] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1233.890883] LustreError: 45046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.905375] LustreError: 45046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1239.011918] LustreError: 45045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.847542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.282572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1247.624844] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1254.915728] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1261.200978] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1267.430085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1267.710464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1268.733084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1273.206753] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1280.041827] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1292.260046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.264621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.277726] Lustre: Skipped 1 previous similar message [ 1292.292691] Lustre: Skipped 2 previous similar messages [ 1293.052285] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (stopping) [ 1296.388688] Lustre: server umount lustre-MDT0000 complete [ 1297.378829] LustreError: 45773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.413887] LustreError: 45773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1300.182327] LustreError: 45026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628369 with bad export cookie 15608430573599236746 [ 1300.192510] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.202586] LustreError: 45026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1302.511262] LustreError: 47540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1302.512220] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.531465] LustreError: 47540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1302.574032] Lustre: Skipped 1 previous similar message [ 1302.578126] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1305.506931] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1305.513532] Lustre: Skipped 1 previous similar message [ 1306.585266] Lustre: server umount lustre-MDT0001 complete [ 1335.263147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1335.450312] Lustre: server umount lustre-OST0000 complete [ 1351.662557] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1354.473919] Key type lgssc unregistered [ 1354.787607] LNet: 48912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.796044] LNetError: 48912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.817747] LNet: Removed LNI 192.168.203.157@tcp [ 1356.430861] Key type .llcrypt unregistered [ 1356.432679] Key type ._llcrypt unregistered [ 1383.966045] Key type ._llcrypt registered [ 1383.969267] Key type .llcrypt registered [ 1384.102168] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1384.794173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1384.809059] alg: No test for adler32 (adler32-zlib) [ 1385.876370] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1386.193347] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1387.847288] Key type lgssc registered [ 1388.968793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1395.456307] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1395.489539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.039371] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1416.159429] LustreError: 50298:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 1442.847347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.398387] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1455.228889] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1619.586516] Lustre: server umount lustre-OST0000 complete [ 1636.156658] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1638.918507] Key type lgssc unregistered [ 1639.214663] LNet: 51976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.223794] LNetError: 51976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.243393] LNet: Removed LNI 192.168.203.157@tcp [ 1640.046762] Key type .llcrypt unregistered [ 1640.049130] Key type ._llcrypt unregistered [ 1662.496664] Key type ._llcrypt registered [ 1662.501031] Key type .llcrypt registered [ 1662.596762] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1663.549204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1663.574634] alg: No test for adler32 (adler32-zlib) [ 1664.634630] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1664.828499] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1666.487231] Key type lgssc registered [ 1667.421950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1674.681597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1674.697040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1676.117243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.934381] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1686.504171] LustreError: 53390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.521261] LustreError: 53390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1686.778955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.047707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1690.831309] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1697.272786] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1701.726928] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1707.174559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1707.398390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1712.455258] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1719.066562] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1722.083994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.084733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1722.115566] Lustre: Skipped 1 previous similar message [ 1727.326426] Lustre: server umount lustre-OST0000 complete [ 1732.578759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.596918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1732.602899] Lustre: Skipped 1 previous similar message [ 1736.882933] Lustre: server umount lustre-MDT0000 complete [ 1737.697040] LustreError: 54129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.712361] LustreError: 54129:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1740.139133] LustreError: 53371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628809 with bad export cookie 8072791040126581021 [ 1740.148237] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1740.150347] LustreError: 53371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1740.429273] Lustre: server umount lustre-MDT0001 complete [ 1749.805731] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1752.357816] Key type lgssc unregistered [ 1752.650436] LNet: 56923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.661083] LNetError: 56923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.684213] LNet: Removed LNI 192.168.203.157@tcp [ 1753.475543] Key type .llcrypt unregistered [ 1753.477956] Key type ._llcrypt unregistered [ 1774.868478] Key type ._llcrypt registered [ 1774.874129] Key type .llcrypt registered [ 1774.983390] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1776.312918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1776.389136] alg: No test for adler32 (adler32-zlib) [ 1777.449967] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1777.749941] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1779.423148] Key type lgssc registered [ 1780.530225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.802446] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1787.841384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1803.360623] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1808.479271] LustreError: 58309:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 1835.103368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.395402] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1847.963171] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1853.900489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.353574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.440301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1858.828062] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1859.553932] LustreError: 58335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1859.588073] LustreError: 58335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1864.674643] LustreError: 58334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1867.545136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1867.946698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1872.031473] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1879.460693] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1884.525620] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1888.737773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1888.744786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1888.763059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1889.774889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.776017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1889.795414] Lustre: Skipped 1 previous similar message [ 1892.653879] Lustre: server umount lustre-OST0000 complete [ 1913.829569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1913.840676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.849359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1919.069243] Lustre: server umount lustre-MDT0000 complete [ 1920.489074] LustreError: 59345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1920.508987] LustreError: 59345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1922.578068] LustreError: 58317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786628991 with bad export cookie 8774752862497579979 [ 1922.582424] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.596476] LustreError: 58317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1928.954986] Lustre: server umount lustre-MDT0001 complete [ 1939.024295] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 1941.224578] Key type lgssc unregistered [ 1941.464301] LNet: 62054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.472802] LNetError: 62054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.500325] LNet: Removed LNI 192.168.203.157@tcp [ 1942.197219] Key type .llcrypt unregistered [ 1942.199936] Key type ._llcrypt unregistered [ 1964.102489] Key type ._llcrypt registered [ 1964.105117] Key type .llcrypt registered [ 1964.195879] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 1964.909392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.924076] alg: No test for adler32 (adler32-zlib) [ 1965.929316] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 1966.135813] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 1967.767180] Key type lgssc registered [ 1968.857683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1975.433918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1975.461567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.057775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.711278] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 1987.552649] LustreError: 63467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.577805] LustreError: 63467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1991.449292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.770623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1996.046048] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2003.598383] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2008.958575] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2016.632816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2017.207670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.361749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2024.080701] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2033.198783] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2052.067141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2052.079609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.089151] Lustre: Skipped 1 previous similar message [ 2052.094218] LustreError: 65290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.317700] Lustre: server umount lustre-OST0000 complete [ 2057.185698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.198088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.201103] Lustre: Skipped 1 previous similar message [ 2057.214431] Lustre: Skipped 1 previous similar message [ 2062.305586] LustreError: 63466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.320574] LustreError: 63466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2062.363500] Lustre: server umount lustre-MDT0000 complete [ 2066.129562] LustreError: 63447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629135 with bad export cookie 2674389954635419582 [ 2066.138337] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2066.145436] LustreError: 63447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2066.660658] Lustre: server umount lustre-MDT0001 complete [ 2075.884043] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2078.290478] Key type lgssc unregistered [ 2078.599546] LNet: 67025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2078.607854] LNetError: 67025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2078.623370] LNet: Removed LNI 192.168.203.157@tcp [ 2079.511253] Key type .llcrypt unregistered [ 2079.515193] Key type ._llcrypt unregistered [ 2106.973214] Key type ._llcrypt registered [ 2106.975560] Key type .llcrypt registered [ 2107.100050] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2107.988339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2108.035191] alg: No test for adler32 (adler32-zlib) [ 2109.052587] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2109.244094] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2110.871723] Key type lgssc registered [ 2111.694711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2119.989358] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 09:53:07 (1786629187) [ 2127.901559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2127.914637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.222525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.446345] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2139.629815] LustreError: 68729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.641767] LustreError: 68729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2142.023722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.382665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2146.486913] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2156.308953] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2164.308456] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2171.428372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2171.810839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.895788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2177.628726] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2184.289935] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2192.741887] Lustre: Failing over lustre-MDT0000 [ 2192.996413] Lustre: server umount lustre-MDT0000 complete [ 2193.377560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2193.377954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.380248] LustreError: 68725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.415506] Lustre: Skipped 2 previous similar messages [ 2196.232194] LustreError: 68724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2196.249822] LustreError: 68724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2198.500449] LustreError: 70054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2198.524153] LustreError: 70054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2201.097933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.202295] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2201.325101] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.57@tcp (not set up) [ 2201.353959] Lustre: *** cfs_fail_loc=135, val=0*** [ 2201.462776] LustreError: 71814:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2201.472207] LustreError: 71814:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -2 [ 2201.481529] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2201.481529] [ 2201.494094] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2201.511156] LustreError: 71803:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2201.523986] LustreError: 71803:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2201.539243] LustreError: 71803:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2201.549080] LustreError: 71803:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2201.580559] Lustre: server umount lustre-MDT0000 complete [ 2201.586727] LustreError: 71803:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2203.623475] LustreError: 68725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.641532] LustreError: 68725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2207.714567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.855328] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2207.864330] LustreError: 68721:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2207.869344] LustreError: 72159:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.157@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000164aabbb) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2208.119147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2208.165423] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2211.561105] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2212.312208] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2213.351619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2213.395267] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2213.456660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2223.586871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.611267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2223.620129] Lustre: Skipped 1 previous similar message [ 2226.674958] Lustre: server umount lustre-OST0000 complete [ 2233.827627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.831730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.833692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2233.844368] Lustre: Skipped 2 previous similar messages [ 2233.869036] Lustre: Skipped 1 previous similar message [ 2236.320333] Lustre: server umount lustre-MDT0000 complete [ 2238.949923] LustreError: 68725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.977786] LustreError: 68725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2239.894235] LustreError: 69464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629308 with bad export cookie 11472715624901698739 [ 2239.900236] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.902618] LustreError: 69464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2240.273426] Lustre: server umount lustre-MDT0001 complete [ 2250.662934] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2253.559045] Key type lgssc unregistered [ 2253.854702] LNet: 73597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.867789] LNetError: 73597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2254.890629] LNet: Removed LNI 192.168.203.157@tcp [ 2255.796523] Key type .llcrypt unregistered [ 2255.798686] Key type ._llcrypt unregistered [ 2279.394184] Key type ._llcrypt registered [ 2279.396653] Key type .llcrypt registered [ 2279.504444] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2280.595230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.614938] alg: No test for adler32 (adler32-zlib) [ 2281.685887] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2281.920376] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2283.591968] Key type lgssc registered [ 2284.628141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2292.230257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2292.276261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.882064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.873660] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2303.465624] LustreError: 75020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.494340] LustreError: 75020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2306.335943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.604617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2311.336869] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2318.714677] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2324.619679] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2331.655714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2332.110095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.239938] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2338.286981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2347.549823] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2356.543888] Lustre: Failing over lustre-MDT0000 [ 2356.710439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2356.722162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2356.731607] Lustre: Skipped 2 previous similar messages [ 2356.759881] Lustre: Skipped 2 previous similar messages [ 2356.925243] Lustre: server umount lustre-MDT0000 complete [ 2359.556092] LustreError: 75014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.826272] LustreError: 75015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.858458] LustreError: 75015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2364.653462] LustreError: 77144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.517551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.626725] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2366.794198] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2366.927472] LustreError: 78108:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2366.938516] LustreError: 78108:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -9 [ 2366.951980] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2366.951980] [ 2366.961865] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2366.979060] LustreError: 78096:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2366.988254] LustreError: 78096:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2366.994027] LustreError: 78096:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2367.001740] LustreError: 78096:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2367.032097] Lustre: server umount lustre-MDT0000 complete [ 2367.043128] LustreError: 78096:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2369.776967] LustreError: 75016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2369.791963] LustreError: 75016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2373.988372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.173638] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2374.440336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2374.508475] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2374.895245] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2379.338833] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2379.768044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2379.795555] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2379.866817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2387.428128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2387.435093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.448888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2389.989961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2389.993666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.111426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2395.120267] Lustre: Skipped 2 previous similar messages [ 2400.240425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2400.246363] Lustre: Skipped 1 previous similar message [ 2401.247151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2401.438539] Lustre: server umount lustre-OST0000 complete [ 2410.465044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.478257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2410.479764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2410.501404] Lustre: Skipped 1 previous similar message [ 2411.833634] Lustre: server umount lustre-MDT0000 complete [ 2415.420081] LustreError: 74999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629484 with bad export cookie 13419320164257450812 [ 2415.426563] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2415.428810] LustreError: 74999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2415.706653] Lustre: server umount lustre-MDT0001 complete [ 2426.491354] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2428.710273] Key type lgssc unregistered [ 2428.908118] LNet: 79893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2428.918661] LNetError: 79893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.990624] LNet: Removed LNI 192.168.203.157@tcp [ 2430.642547] Key type .llcrypt unregistered [ 2430.644802] Key type ._llcrypt unregistered [ 2441.462529] Key type ._llcrypt registered [ 2441.465464] Key type .llcrypt registered [ 2441.575310] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2441.825408] Key type .llcrypt unregistered [ 2441.830431] Key type ._llcrypt unregistered [ 2464.212603] Key type ._llcrypt registered [ 2464.216079] Key type .llcrypt registered [ 2464.311343] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2465.147632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2465.164150] alg: No test for adler32 (adler32-zlib) [ 2466.295476] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2466.506390] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2468.199277] Key type lgssc registered [ 2469.639041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2477.006949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2477.040573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.462748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.897731] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2488.814952] LustreError: 81801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.843317] LustreError: 81801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2490.824282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.268891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2495.383650] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2502.378342] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2507.699313] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2513.567791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2513.947849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.317918] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2523.218502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2528.268560] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2538.478933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.484212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2538.499425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2543.587519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2543.594684] Lustre: Skipped 2 previous similar messages [ 2544.137764] Lustre: server umount lustre-OST0000 complete [ 2547.682159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2547.692461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.709742] Lustre: Skipped 1 previous similar message [ 2547.716493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2548.709033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.467409] Lustre: server umount lustre-MDT0000 complete [ 2553.845568] LustreError: 81800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.864209] LustreError: 81800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2556.867521] LustreError: 81779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629625 with bad export cookie 11013936734171615524 [ 2556.873733] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2556.881110] LustreError: 81779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2557.262158] Lustre: server umount lustre-MDT0001 complete [ 2566.717532] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2569.452584] Key type lgssc unregistered [ 2569.707377] LNet: 85333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2569.714987] LNetError: 85333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2569.733663] LNet: Removed LNI 192.168.203.157@tcp [ 2570.612395] Key type .llcrypt unregistered [ 2570.616113] Key type ._llcrypt unregistered [ 2591.619593] Key type ._llcrypt registered [ 2591.623556] Key type .llcrypt registered [ 2591.729658] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2592.952928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2593.091819] alg: No test for adler32 (adler32-zlib) [ 2594.125333] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2594.399470] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2596.095159] Key type lgssc registered [ 2597.192113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2604.328892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2604.344318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.745058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.363366] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2615.274689] LustreError: 86757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.294548] LustreError: 86757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2617.504414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.857985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2621.749373] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2628.881503] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2635.393317] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2642.829649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2643.357963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.642768] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2650.544825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2656.531361] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2665.954058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2665.962057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.970390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2668.002195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2668.006184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2668.013189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.049655] Lustre: server umount lustre-OST0000 complete [ 2678.248949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.253759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2678.267415] Lustre: Skipped 1 previous similar message [ 2678.286761] Lustre: Skipped 2 previous similar messages [ 2682.114109] Lustre: server umount lustre-MDT0000 complete [ 2683.370391] LustreError: 87806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.404650] LustreError: 87806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2685.990754] LustreError: 86738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629755 with bad export cookie 3902555854374406419 [ 2685.999598] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2686.002229] LustreError: 86738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2686.404182] Lustre: server umount lustre-MDT0001 complete [ 2696.589729] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2699.599429] Key type lgssc unregistered [ 2699.896385] LNet: 90295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2699.903205] LNetError: 90295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2699.925936] LNet: Removed LNI 192.168.203.157@tcp [ 2700.874084] Key type .llcrypt unregistered [ 2700.880329] Key type ._llcrypt unregistered [ 2726.148359] Key type ._llcrypt registered [ 2726.151626] Key type .llcrypt registered [ 2726.317611] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2727.325828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2727.398951] alg: No test for adler32 (adler32-zlib) [ 2728.665434] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2728.900042] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2730.623175] Key type lgssc registered [ 2731.701291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2738.916290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2738.960524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2740.576480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.205208] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2749.923860] LustreError: 91707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.946869] LustreError: 91707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2752.955293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.265439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2757.825416] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2765.001176] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2770.928951] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2778.042137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2778.403248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2784.465762] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2785.583950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 2791.030761] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2814.434701] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2814.439156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.455913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2814.465128] Lustre: Skipped 1 previous similar message [ 2819.556536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2819.574137] Lustre: Skipped 1 previous similar message [ 2824.677443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2824.688486] Lustre: Skipped 1 previous similar message [ 2826.210067] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2826.461936] Lustre: server umount lustre-OST0000 complete [ 2834.920836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.925718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2834.948663] Lustre: Skipped 2 previous similar messages [ 2834.973939] Lustre: Skipped 1 previous similar message [ 2837.016755] Lustre: server umount lustre-MDT0000 complete [ 2840.035147] LustreError: 91702:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2840.048502] LustreError: 91702:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2840.592685] LustreError: 91688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786629909 with bad export cookie 9264612097463855584 [ 2840.606163] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2840.608192] LustreError: 91688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2840.931086] Lustre: server umount lustre-MDT0001 complete [ 2852.296927] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 2854.890687] Key type lgssc unregistered [ 2855.214577] LNet: 95268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2855.233705] LNetError: 95268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2855.260419] LNet: Removed LNI 192.168.203.157@tcp [ 2856.134311] Key type .llcrypt unregistered [ 2856.136746] Key type ._llcrypt unregistered [ 2878.759787] Key type ._llcrypt registered [ 2878.766300] Key type .llcrypt registered [ 2878.858618] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 2879.840969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2879.865955] alg: No test for adler32 (adler32-zlib) [ 2880.914824] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 2881.143960] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 2882.815192] Key type lgssc registered [ 2884.064287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2891.535466] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2891.554975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.103403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2912.223808] LustreError: 96664:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 2938.847175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2944.681099] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2950.961750] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2956.524189] Lustre: server umount lustre-OST0000 complete [ 2965.783904] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:07:13 (1786630033) [ 2972.861327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.488260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2977.883387] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2978.805294] LustreError: 98302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2978.835202] LustreError: 98302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2983.905537] LustreError: 98303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2986.131455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2986.358931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2989.964456] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 2996.731979] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3003.083036] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3010.914598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3011.198923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3017.654548] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3018.367082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3025.733283] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3042.278571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3042.293831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3042.316196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3046.661906] Lustre: server umount lustre-OST0000 complete [ 3052.517157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3052.528944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3052.532549] Lustre: Skipped 2 previous similar messages [ 3052.549568] Lustre: Skipped 2 previous similar messages [ 3056.898425] Lustre: server umount lustre-MDT0000 complete [ 3057.632926] LustreError: 98299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.665351] LustreError: 98299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3060.667929] LustreError: 98284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630129 with bad export cookie 8822556833744892693 [ 3060.677778] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3060.693187] LustreError: 98284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3061.084680] Lustre: server umount lustre-MDT0001 complete [ 3071.299994] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3073.775231] Key type lgssc unregistered [ 3074.054964] LNet: 101996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3074.064748] LNetError: 101996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3075.117042] LNet: Removed LNI 192.168.203.157@tcp [ 3075.977644] Key type .llcrypt unregistered [ 3075.983049] Key type ._llcrypt unregistered [ 3116.317399] Key type ._llcrypt registered [ 3116.318934] Key type .llcrypt registered [ 3116.419072] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3117.293542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3117.352090] alg: No test for adler32 (adler32-zlib) [ 3118.299194] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 3118.533441] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3120.215502] Key type lgssc registered [ 3121.110624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3129.206242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3129.245647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.040220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3136.128628] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3141.605756] LustreError: 104184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.617820] LustreError: 104184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3144.723298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.234315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3149.005927] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3158.493715] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3166.006712] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3174.986620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3175.257049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.296729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3181.125296] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3188.546608] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3200.996955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3201.014520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3201.015607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3201.028307] Lustre: Skipped 1 previous similar message [ 3203.192219] Lustre: server umount lustre-OST0000 complete [ 3206.116433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.122460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.133425] Lustre: Skipped 2 previous similar messages [ 3211.232669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3211.242361] Lustre: Skipped 2 previous similar messages [ 3212.344490] Lustre: server umount lustre-MDT0000 complete [ 3215.428588] LustreError: 104163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630284 with bad export cookie 15151728952507563847 [ 3215.434852] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3215.440789] LustreError: 104163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3215.782506] Lustre: server umount lustre-MDT0001 complete [ 3224.579977] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3227.360875] Key type lgssc unregistered [ 3227.719150] LNet: 107722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3227.734455] LNetError: 107722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3227.759585] LNet: Removed LNI 192.168.203.157@tcp [ 3228.541320] Key type .llcrypt unregistered [ 3228.545343] Key type ._llcrypt unregistered [ 3245.667717] Key type ._llcrypt registered [ 3245.671616] Key type .llcrypt registered [ 3245.810897] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3246.943226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3246.965587] alg: No test for adler32 (adler32-zlib) [ 3247.982485] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 3248.163829] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3249.839643] Key type lgssc registered [ 3250.718894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3257.528516] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3257.558807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3273.120868] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3278.239321] LustreError: 108958:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 3304.928540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.691530] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3316.721645] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3322.514396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.787691] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3322.796573] LustreError: 109981:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3322.802783] LustreError: 109981:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3322.810826] LustreError: 109981:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3322.876321] Lustre: server umount lustre-MDT0000 complete [ 3322.878873] LustreError: 109981:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3339.383121] Lustre: server umount lustre-OST0000 complete [ 3348.340689] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 3357.818044] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3391.357906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3391.562941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3391.580523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3391.655239] Lustre: lustre-MDT0000: new disk, initializing [ 3391.761631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.782362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3395.862700] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3405.089852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.201739] Lustre: 112771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3405.247307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3405.251115] Lustre: Skipped 1 previous similar message [ 3405.302969] Lustre: lustre-MDT0001: new disk, initializing [ 3405.365793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3405.395779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3405.403975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3408.822789] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3416.841408] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3421.731037] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3427.275056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3427.500315] Lustre: lustre-OST0000: new disk, initializing [ 3427.503899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3427.511546] Lustre: 114739:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3427.592934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3433.717981] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3437.086278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3437.100584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3437.171295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3441.599393] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3446.369913] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3446.566743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3450.497591] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3450.686876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3452.386123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3452.387738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3452.393985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3452.402654] Lustre: Skipped 1 previous similar message [ 3452.413630] Lustre: Skipped 1 previous similar message [ 3457.506495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3457.511495] Lustre: Skipped 1 previous similar message [ 3458.489835] Lustre: server umount lustre-OST0000 complete [ 3462.624978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.643682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3462.652467] Lustre: Skipped 1 previous similar message [ 3467.746104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3467.752604] Lustre: Skipped 1 previous similar message [ 3468.468851] Lustre: server umount lustre-MDT0000 complete [ 3472.322698] LustreError: 112762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630541 with bad export cookie 5311901480610942644 [ 3472.324943] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3472.332366] LustreError: 112762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3472.586333] Lustre: server umount lustre-MDT0001 complete [ 3482.918880] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:15:49 (1786630549) [ 3487.199769] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3503.183801] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 3511.263843] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3545.732334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.953758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3545.974763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3546.062895] Lustre: lustre-MDT0000: new disk, initializing [ 3546.146661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.165708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3550.018631] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3559.494551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.623233] Lustre: 119652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3559.695803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3559.699463] Lustre: Skipped 1 previous similar message [ 3559.775950] Lustre: lustre-MDT0001: new disk, initializing [ 3559.843535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3559.879674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3559.900663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3563.354614] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3572.669581] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3581.191448] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3591.382235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3591.560452] Lustre: lustre-OST0000: new disk, initializing [ 3591.563081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3591.566807] Lustre: 121620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3591.671259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3598.068045] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3598.438029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3598.446784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3598.593497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3607.777669] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3612.016513] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3612.194782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3616.273109] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3616.423453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3619.304936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3619.317301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3619.326828] LustreError: Skipped 1 previous similar message [ 3619.335907] Lustre: Skipped 1 previous similar message [ 3619.344200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3629.537634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3629.544251] Lustre: Skipped 4 previous similar messages [ 3632.096672] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3632.312725] Lustre: server umount lustre-OST0000 complete [ 3636.706165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3636.714443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.757622] Lustre: Skipped 1 previous similar message [ 3641.822615] Lustre: server umount lustre-MDT0000 complete [ 3645.820580] LustreError: 119644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630714 with bad export cookie 5311901480610943526 [ 3645.831078] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3645.849932] LustreError: 119644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3646.432656] Lustre: server umount lustre-MDT0001 complete [ 3655.776954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.423088] LustreError: 123837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3656.444141] LustreError: 123837:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3656.530939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3661.561383] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3661.807631] LustreError: 123838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.916391] LustreError: 123837:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3668.611202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3672.394869] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3679.394427] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3684.826300] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3690.471913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3696.089160] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3703.227995] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3711.869580] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3719.656749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3719.665728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3719.682977] Lustre: Skipped 1 previous similar message [ 3719.691263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3719.699000] Lustre: Skipped 3 previous similar messages [ 3725.300467] Lustre: server umount lustre-OST0000 complete [ 3730.400821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3730.409717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3730.433019] Lustre: Skipped 1 previous similar message [ 3735.865652] Lustre: server umount lustre-MDT0000 complete [ 3738.083778] LustreError: 123838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.111612] LustreError: 123838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3739.624049] LustreError: 123817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630808 with bad export cookie 5311901480610944408 [ 3739.630255] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3739.642745] LustreError: 123817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3739.972393] Lustre: server umount lustre-MDT0001 complete [ 3750.318116] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 3753.518971] Key type lgssc unregistered [ 3753.862288] LNet: 127471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3753.875701] LNetError: 127471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3753.905284] LNet: Removed LNI 192.168.203.157@tcp [ 3754.816892] Key type .llcrypt unregistered [ 3754.819124] Key type ._llcrypt unregistered [ 3774.158959] Key type ._llcrypt registered [ 3774.164279] Key type .llcrypt registered [ 3774.334856] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 3790.888085] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 3791.849245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3791.912389] alg: No test for adler32 (adler32-zlib) [ 3793.047582] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 3793.281801] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 3794.911134] Key type lgssc registered [ 3795.924876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3828.487354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3828.514247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3829.721936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3829.753235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3829.856235] Lustre: lustre-MDT0000: new disk, initializing [ 3829.929933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.950040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3833.769082] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3842.678870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3842.745770] Lustre: 130639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3842.805625] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3842.806692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3842.813558] Lustre: Skipped 1 previous similar message [ 3842.875657] Lustre: lustre-MDT0001: new disk, initializing [ 3842.966079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3843.050421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3843.055709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3846.280610] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3855.559387] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3861.296553] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3868.045724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3868.389483] Lustre: lustre-OST0000: new disk, initializing [ 3868.393624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3868.399781] Lustre: 132610:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3868.530079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3874.742766] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3876.463041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3876.475286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3876.579349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3883.871662] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3887.901665] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3888.126924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3892.215140] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3892.416499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3897.315705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3897.319300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.319968] LustreError: Skipped 1 previous similar message [ 3897.321611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3897.333734] Lustre: Skipped 1 previous similar message [ 3900.317522] Lustre: server umount lustre-OST0000 complete [ 3904.480276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3904.488394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.498299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3904.500994] Lustre: Skipped 1 previous similar message [ 3907.553589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3907.562258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3907.577152] Lustre: Skipped 1 previous similar message [ 3909.382917] Lustre: server umount lustre-MDT0000 complete [ 3912.677093] LustreError: 130633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786630981 with bad export cookie 14659926716063462544 [ 3912.684608] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3912.688408] LustreError: 130633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3913.018299] Lustre: server umount lustre-MDT0001 complete [ 3921.275248] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:23:09 (1786630989) [ 3927.496112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3927.924068] LustreError: 135019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.942843] LustreError: 135019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3928.052503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3932.218310] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3933.152535] LustreError: 135019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.174964] LustreError: 135019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3938.277200] LustreError: 135020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3940.204191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.550071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3945.088847] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 3953.294237] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3960.175886] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3963.367107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.368867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3963.388769] Lustre: Skipped 1 previous similar message [ 3963.419728] Lustre: Skipped 1 previous similar message [ 3969.182683] Lustre: server umount lustre-MDT0000 complete [ 3970.531626] LustreError: 136229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3970.549547] LustreError: 136229:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3973.850367] LustreError: 135001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631042 with bad export cookie 14659926716063463433 [ 3973.856931] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3973.861822] LustreError: 135001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3974.122492] Lustre: server umount lustre-MDT0001 complete [ 3982.258627] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:24:09 (1786631049) [ 3989.082208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4004.706538] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4009.825281] LustreError: 137433:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 4035.615441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4042.655513] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4048.638323] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4050.311690] Lustre: server umount lustre-OST0000 complete [ 4057.483689] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:25:25 (1786631125) [ 4066.158938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.875785] LustreError: 138878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.983024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4072.769475] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4077.539871] LustreError: 138879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.575168] LustreError: 138879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4083.050996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4088.278934] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4096.104128] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4102.599604] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4109.249139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4109.526885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4109.532269] Lustre: Skipped 1 previous similar message [ 4115.367863] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4122.548377] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4137.908408] Lustre: server umount lustre-MDT0000 complete [ 4139.489661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.493441] LustreError: 138874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4139.501533] Lustre: Skipped 1 previous similar message [ 4139.512576] LustreError: 138874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4141.146121] LustreError: 138860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631210 with bad export cookie 14659926716063464098 [ 4141.149627] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4141.156282] LustreError: 138860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4141.429936] Lustre: server umount lustre-MDT0001 complete [ 4154.084421] Lustre: server umount lustre-OST0000 complete [ 4162.172630] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:27:10 (1786631230) [ 4169.289364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.670436] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4174.819451] LustreError: 142557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4174.835562] LustreError: 142557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4183.508413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.886356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4183.891911] Lustre: Skipped 1 previous similar message [ 4189.282665] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4197.954614] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4205.520144] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4212.857263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4214.335419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4219.944266] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4227.166634] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4231.228708] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4231.431720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4235.962509] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4236.147674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4238.822794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.824439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4238.825191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4238.825198] Lustre: Skipped 2 previous similar messages [ 4238.835830] Lustre: Skipped 2 previous similar messages [ 4252.128166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4252.378838] Lustre: server umount lustre-OST0000 complete [ 4259.298774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4259.316044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4259.325740] Lustre: Skipped 1 previous similar message [ 4259.353378] Lustre: Skipped 6 previous similar messages [ 4262.254426] Lustre: server umount lustre-MDT0000 complete [ 4264.417514] LustreError: 142557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4264.435864] LustreError: 142557:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4265.908509] LustreError: 142539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631334 with bad export cookie 14659926716063465162 [ 4265.913096] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4265.917142] LustreError: 142539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4266.137656] Lustre: server umount lustre-MDT0001 complete [ 4274.562042] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:29:02 (1786631342) [ 4282.065383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4297.699787] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4302.815845] LustreError: 146715:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 4328.607308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4328.614506] Lustre: Skipped 1 previous similar message [ 4334.678841] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4341.759868] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4347.915298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.516448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4352.657816] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4360.467517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.123881] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4373.056638] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4378.299662] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4382.651958] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4382.811141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4388.108426] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4388.350498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4391.392274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4391.400162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.408258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4397.134740] Lustre: server umount lustre-MDT0000 complete [ 4399.076533] LustreError: 147751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4399.093622] LustreError: 147751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4401.919860] LustreError: 146723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631470 with bad export cookie 14659926716063466191 [ 4401.938951] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4401.941171] LustreError: 146723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4402.411247] Lustre: server umount lustre-MDT0001 complete [ 4416.545447] Lustre: server umount lustre-OST0000 complete [ 4426.124481] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:31:33 (1786631493) [ 4433.678279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4449.375622] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4454.498584] LustreError: 150912:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 4487.118837] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4495.572857] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4503.031343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.121917] LustreError: 128864:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000051e8527f x1873418242907520/t0(0) o250->MGC192.168.203.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4503.544024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4508.427918] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4517.890206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4522.697190] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4531.569466] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4537.796532] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4545.339625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4545.529406] Lustre: lustre-OST0001: new disk, initializing [ 4545.534133] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4545.541832] Lustre: 153786:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4551.234802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4551.249286] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4551.365166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4551.932902] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4560.708949] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4565.336192] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4565.512995] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4570.220434] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4570.486966] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4574.689946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4574.703298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4574.723633] Lustre: Skipped 2 previous similar messages [ 4574.737348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4574.743969] Lustre: Skipped 3 previous similar messages [ 4578.954679] Lustre: server umount lustre-OST0000 complete [ 4587.496096] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4587.504357] LustreError: Skipped 1 previous similar message [ 4589.411398] Lustre: server umount lustre-OST0001 complete [ 4600.426062] Lustre: server umount lustre-MDT0000 complete [ 4605.071799] LustreError: 150918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631674 with bad export cookie 14659926716063467143 [ 4605.076339] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4605.082921] LustreError: 150918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4605.406982] Lustre: server umount lustre-MDT0001 complete [ 4632.175293] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:34:59 (1786631699) [ 4633.729953] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4635.829860] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4637.552905] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:35:05 (1786631705) [ 4643.410810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.660130] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4643.683224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4643.822797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4643.825865] Lustre: Skipped 6 previous similar messages [ 4648.635785] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4657.822484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.956982] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4657.978240] Lustre: 157347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4663.007339] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4671.412291] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4678.061090] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4684.241856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4684.451948] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4691.589657] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4693.724808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4703.607828] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4709.270226] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4709.484910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4715.790660] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4715.997814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4719.074429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4719.086295] LustreError: Skipped 2 previous similar messages [ 4719.089828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.097714] Lustre: Skipped 5 previous similar messages [ 4719.101464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4719.112316] Lustre: Skipped 8 previous similar messages [ 4724.468210] Lustre: server umount lustre-OST0000 complete [ 4747.714300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4755.655990] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4759.487955] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4759.505660] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4759.510442] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4759.550446] Lustre: Skipped 1 previous similar message [ 4765.447357] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4771.339978] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4771.573324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4776.062741] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4776.227435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4783.110380] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c52c46ee000.ost_server_uuid 50 [ 4784.904743] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8c52c46ee000.ost_server_uuid in FULL state after 0 sec [ 4785.167582] LustreError: 157354:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.57@tcp) returned error from blocking AST (req@0000000095aaca8e x1873418243021952 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000ae823ad6/0xcb7285ad3838cfe1 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.57@tcp remote: 0xcd3841abf05c96b9 expref: 5 pid: 159217 timeout: 4885 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4785.218433] LustreError: lustre-MDT0000: A client on nid 192.168.203.57@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4785.238212] LustreError: 157343:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.57@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000ae823ad6/0xcb7285ad3838cfe1 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.57@tcp remote: 0xcd3841abf05c96b9 expref: 6 pid: 159217 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4790.241895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4790.256905] LustreError: Skipped 1 previous similar message [ 4796.503059] Lustre: server umount lustre-OST0000 complete [ 4807.159501] Lustre: server umount lustre-MDT0000 complete [ 4810.722432] LustreError: 158258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.752140] LustreError: 158258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 4812.511333] LustreError: 159209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786631881 with bad export cookie 14659926716063468319 [ 4812.536494] LustreError: 159209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4812.544449] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4819.048805] Lustre: server umount lustre-MDT0001 complete [ 4828.644275] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 4831.266960] Key type lgssc unregistered [ 4831.552836] LNet: 163468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4831.561601] LNetError: 163468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4831.591315] LNet: Removed LNI 192.168.203.157@tcp [ 4832.476463] Key type .llcrypt unregistered [ 4832.479949] Key type ._llcrypt unregistered [ 4857.794626] Key type ._llcrypt registered [ 4857.799250] Key type .llcrypt registered [ 4857.919863] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 4858.953895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4858.979426] alg: No test for adler32 (adler32-zlib) [ 4860.109218] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 4860.370586] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 4862.023176] Key type lgssc registered [ 4863.352858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4871.240891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4871.281426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.759777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4877.332568] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4882.918445] LustreError: 164881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4882.943868] LustreError: 164881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4886.513275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.804906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4891.745981] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4900.302360] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4907.289427] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4915.429408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4915.777754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.838684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4921.813406] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4931.809707] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4939.117751] Lustre: Failing over lustre-MDT0000 [ 4939.414552] Lustre: server umount lustre-MDT0000 complete [ 4940.074865] LustreError: 164878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.293892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.303290] LustreError: 164876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.307619] Lustre: Skipped 2 previous similar messages [ 4941.364710] LustreError: 164876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4946.404267] LustreError: 164877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4946.434370] LustreError: 164877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4951.523284] LustreError: 164878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4951.555068] LustreError: 164878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4952.522121] LustreError: 164860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.57@tcp arrived at 1786632021 with bad export cookie 14200431598542162319 [ 4957.669878] Lustre: 164191:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632010/real 1786632010] req@0000000032fbbd95 x1873419361611520/t0(0) o400->MGC192.168.203.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786632026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4957.726848] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4960.893743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4962.786873] LustreError: 164877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4962.822708] LustreError: 164877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4968.175271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4968.229960] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4972.836647] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 4973.541047] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4973.552181] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4974.721294] Lustre: lustre-MDT0000: Denying connection for new client 8adae895-f6d0-41c9-b917-2a0959e628b3 (at 192.168.203.57@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 4979.948204] Lustre: lustre-MDT0000: Denying connection for new client 8adae895-f6d0-41c9-b917-2a0959e628b3 (at 192.168.203.57@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 4983.777530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4983.779431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4983.806429] Lustre: Skipped 1 previous similar message [ 4988.901372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4988.907241] Lustre: Skipped 2 previous similar messages [ 4990.053188] Lustre: server umount lustre-OST0000 complete [ 4993.941203] LustreError: 168616:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4993.953670] Lustre: 167964:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4993.966486] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4993.984172] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4994.000039] LustreError: 167964:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c41e4789 x1873419361631744/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4994.017051] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.045306] LustreError: 167964:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 4994.056628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4994.091033] LustreError: 167964:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 4994.109062] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 4994.564181] Lustre: server umount lustre-MDT0000 complete [ 4998.025749] LustreError: 168414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632067 with bad export cookie 14200431598542162417 [ 4998.029344] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4998.034278] LustreError: 168414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4998.034825] LustreError: 164190:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5004.334072] Lustre: server umount lustre-MDT0001 complete [ 5015.448543] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5018.345610] Key type lgssc unregistered [ 5018.655733] LNet: 169371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5018.665533] LNetError: 169371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5019.689477] LNet: Removed LNI 192.168.203.157@tcp [ 5020.656866] Key type .llcrypt unregistered [ 5020.660106] Key type ._llcrypt unregistered [ 5045.882624] Key type ._llcrypt registered [ 5045.885554] Key type .llcrypt registered [ 5045.984806] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5046.769695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5046.787028] alg: No test for adler32 (adler32-zlib) [ 5047.837432] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5048.076308] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5049.743234] Key type lgssc registered [ 5050.796530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5058.065665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5058.095371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.649460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.301156] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5069.797184] LustreError: 170785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.820554] LustreError: 170785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5073.509876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.035805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5078.864325] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5086.291141] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5092.052801] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5098.559820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5098.886345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.989775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5105.177420] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5113.485932] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5119.464185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.492431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5119.497516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5119.534875] Lustre: Skipped 1 previous similar message [ 5124.579852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5124.589191] Lustre: Skipped 1 previous similar message [ 5125.161359] Lustre: server umount lustre-OST0000 complete [ 5129.698176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5129.712147] Lustre: Skipped 1 previous similar message [ 5129.718392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5129.724850] Lustre: Skipped 1 previous similar message [ 5134.816486] LustreError: 170785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5134.830513] LustreError: 170785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5134.920764] Lustre: server umount lustre-MDT0000 complete [ 5138.691208] LustreError: 170766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632207 with bad export cookie 445504989496641547 [ 5138.711462] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5138.977721] Lustre: server umount lustre-MDT0001 complete [ 5149.744138] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5152.652541] Key type lgssc unregistered [ 5152.862451] LNet: 174321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5152.870766] LNetError: 174321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5152.887820] LNet: Removed LNI 192.168.203.157@tcp [ 5153.548369] Key type .llcrypt unregistered [ 5153.553619] Key type ._llcrypt unregistered [ 5187.408824] Key type ._llcrypt registered [ 5187.413462] Key type .llcrypt registered [ 5187.564643] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5188.903920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5188.921496] alg: No test for adler32 (adler32-zlib) [ 5189.967508] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5190.186668] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5191.847321] Key type lgssc registered [ 5193.000709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5200.198762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5200.229360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5201.603962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5206.311136] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5211.112151] LustreError: 176027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5211.144922] LustreError: 176027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5215.007925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.536099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5220.211727] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5231.525871] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5239.501608] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5248.362470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5248.796696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5255.457392] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5263.352932] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5271.340043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5285.860126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5285.879498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.917607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5286.880266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5286.885665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.885922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5288.465409] Lustre: server umount lustre-OST0000 complete [ 5297.121838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.148628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5297.162889] Lustre: Skipped 2 previous similar messages [ 5299.702397] Lustre: server umount lustre-MDT0000 complete [ 5302.241809] LustreError: 177043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5302.249321] LustreError: 177043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5303.312675] LustreError: 176007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632372 with bad export cookie 2316012212496304208 [ 5303.318849] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.343577] LustreError: 176007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5303.764571] Lustre: server umount lustre-MDT0001 complete [ 5315.531539] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5318.336675] Key type lgssc unregistered [ 5318.601727] LNet: 179587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5318.616397] LNetError: 179587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5318.640953] LNet: Removed LNI 192.168.203.157@tcp [ 5319.503606] Key type .llcrypt unregistered [ 5319.507170] Key type ._llcrypt unregistered [ 5342.013702] Key type ._llcrypt registered [ 5342.015730] Key type .llcrypt registered [ 5342.102783] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5342.974255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5343.025708] alg: No test for adler32 (adler32-zlib) [ 5344.071348] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5344.352877] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5346.063264] Key type lgssc registered [ 5347.184700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5357.974495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5357.999101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5359.494607] Lustre: *** cfs_fail_loc=135, val=0*** [ 5359.614025] LustreError: 181111:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5359.622171] LustreError: 181111:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.157@tcp: cfg command failed: rc = -2 [ 5359.631300] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5359.631300] [ 5359.640044] LustreError: MGC192.168.203.157@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5359.655581] LustreError: 181071:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5359.663539] LustreError: 181071:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5359.669714] LustreError: 181071:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5359.675462] LustreError: 181071:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5359.771667] Lustre: server umount lustre-MDT0000 complete [ 5359.775798] LustreError: 181071:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5366.616798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.122915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5371.760589] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5377.506928] LustreError: 181483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5377.518383] LustreError: 181483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5379.732563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5380.178264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5385.092301] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5393.148251] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5398.744460] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5402.592356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5402.598713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5402.604017] Lustre: Skipped 1 previous similar message [ 5402.617863] Lustre: Skipped 1 previous similar message [ 5405.665822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5405.671859] Lustre: Skipped 1 previous similar message [ 5410.801735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5410.808602] Lustre: Skipped 1 previous similar message [ 5415.391188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5415.580523] Lustre: server umount lustre-MDT0000 complete [ 5415.915406] LustreError: 182211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5415.947211] LustreError: 182211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5418.903991] LustreError: 181464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632487 with bad export cookie 9804484613575292700 [ 5418.916380] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5418.920052] LustreError: 181464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5419.349928] Lustre: server umount lustre-MDT0001 complete [ 5432.599968] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5435.575072] Key type lgssc unregistered [ 5435.863247] LNet: 183808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5435.873208] LNetError: 183808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5435.898032] LNet: Removed LNI 192.168.203.157@tcp [ 5436.849326] Key type .llcrypt unregistered [ 5436.853856] Key type ._llcrypt unregistered [ 5461.456839] Key type ._llcrypt registered [ 5461.459764] Key type .llcrypt registered [ 5461.632096] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5461.998766] Key type .llcrypt unregistered [ 5462.001608] Key type ._llcrypt unregistered [ 5479.120695] Key type ._llcrypt registered [ 5479.123207] Key type .llcrypt registered [ 5479.207447] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5480.534577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5480.572577] alg: No test for adler32 (adler32-zlib) [ 5481.723547] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5482.031811] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5483.735211] Key type lgssc registered [ 5484.928417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5492.726233] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5492.745761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5508.255403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5513.375546] LustreError: 185945:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.157@tcp: failed processing log, type 4: rc = -110 [ 5540.064649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5546.696264] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5555.296986] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5562.553359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5563.064592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5563.125540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5567.872806] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5568.486060] LustreError: 185970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5568.505806] LustreError: 185970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5573.601626] LustreError: 185971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.637639] LustreError: 186985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.659843] LustreError: 186985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5576.484937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5576.912862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5581.725241] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5589.770609] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5595.639398] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5598.550031] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5613.032465] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5613.034196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.046692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5613.074225] Lustre: Skipped 1 previous similar message [ 5616.373198] Lustre: server umount lustre-OST0000 complete [ 5623.269314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5623.270316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5623.283492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5623.283507] Lustre: Skipped 1 previous similar message [ 5623.321770] Lustre: Skipped 1 previous similar message [ 5626.654588] Lustre: server umount lustre-MDT0000 complete [ 5628.387456] LustreError: 186988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5628.410476] LustreError: 186988:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5630.347878] LustreError: 185952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632699 with bad export cookie 11069943435079118269 [ 5630.353369] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5630.365592] LustreError: 185952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5630.652696] Lustre: server umount lustre-MDT0001 complete [ 5640.785911] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5644.023260] Key type lgssc unregistered [ 5644.349603] LNet: 189964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5644.362171] LNetError: 189964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5644.384783] LNet: Removed LNI 192.168.203.157@tcp [ 5645.304150] Key type .llcrypt unregistered [ 5645.311994] Key type ._llcrypt unregistered [ 5671.262477] Key type ._llcrypt registered [ 5671.264735] Key type .llcrypt registered [ 5671.419687] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5672.681436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5672.805129] alg: No test for adler32 (adler32-zlib) [ 5673.986694] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5674.222443] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5675.895128] Key type lgssc registered [ 5677.229632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5686.088118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5686.115084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.636431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5692.322941] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5696.997874] LustreError: 191376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5697.016341] LustreError: 191376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5700.710797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.130748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5705.207642] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5714.368980] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5722.197242] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5730.572626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5730.771169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5732.857916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5737.468742] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5747.012709] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5752.472342] Lustre: Failing over lustre-MDT0000 [ 5754.114424] LustreError: 191370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5754.711789] Lustre: server umount lustre-MDT0000 complete [ 5756.391423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.398044] LustreError: 191371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5756.401489] Lustre: Skipped 2 previous similar messages [ 5756.418891] LustreError: 191371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5759.212518] LustreError: 193222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.228607] LustreError: 193222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5764.329568] LustreError: 192413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5764.353246] LustreError: 192413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5772.768718] Lustre: 190687:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786632825/real 1786632825] req@00000000208d539a x1873420215152128/t0(0) o400->MGC192.168.203.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786632841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5772.825347] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5772.856673] LustreError: 192415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5772.898087] LustreError: 192415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5774.916365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.990820] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x18f5a7ea2f93a18 [ 5782.018963] LustreError: 194441:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.157@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000005a7ad675) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 5782.035695] Lustre: MGC192.168.203.157@tcp: Connection restored to 0@lo (at 0@lo) [ 5782.288290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5782.338902] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5785.834177] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5786.573956] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5787.626105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5787.650753] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5787.687710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5791.161631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5801.149216] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5808.107888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.112826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.132065] Lustre: Skipped 1 previous similar message [ 5813.223551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.226117] Lustre: Skipped 1 previous similar message [ 5814.053025] Lustre: server umount lustre-OST0000 complete [ 5823.458503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5823.458957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5823.468569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5823.489864] Lustre: Skipped 1 previous similar message [ 5828.578587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5828.595309] Lustre: Skipped 2 previous similar messages [ 5833.183283] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5833.391781] Lustre: server umount lustre-MDT0000 complete [ 5833.697357] LustreError: 192415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5833.711408] LustreError: 192415:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 5838.029662] LustreError: 191356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786632907 with bad export cookie 112408015654173208 [ 5838.049143] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5838.056521] LustreError: 191356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5838.600064] Lustre: server umount lustre-MDT0001 complete [ 5851.526245] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 5854.844247] Key type lgssc unregistered [ 5855.186854] LNet: 196319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5855.196452] LNetError: 196319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5856.232331] LNet: Removed LNI 192.168.203.157@tcp [ 5857.053492] Key type .llcrypt unregistered [ 5857.057122] Key type ._llcrypt unregistered [ 5883.321831] Key type ._llcrypt registered [ 5883.326040] Key type .llcrypt registered [ 5883.504855] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 5884.787913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5884.865136] alg: No test for adler32 (adler32-zlib) [ 5885.975492] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 5886.221537] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 5887.919172] Key type lgssc registered [ 5889.081075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5896.592925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5896.611254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5898.090753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.651513] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5908.450159] LustreError: 197736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5908.464513] LustreError: 197736:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5911.445526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5911.866130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5915.787372] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5923.219286] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5929.795693] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5937.203571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5937.564178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5943.749780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 5945.351653] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 5952.838379] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5956.728175] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 5964.561744] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 5982.456812] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 5997.026457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5997.039824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.059215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6000.097817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6000.102817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6000.152633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6001.312437] Lustre: server umount lustre-OST0000 complete [ 6005.218478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6005.229956] Lustre: Skipped 1 previous similar message [ 6005.234517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6005.240321] Lustre: Skipped 2 previous similar messages [ 6010.336496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6010.351158] Lustre: Skipped 1 previous similar message [ 6011.415453] Lustre: server umount lustre-MDT0000 complete [ 6015.040626] LustreError: 199587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633084 with bad export cookie 683326459958271609 [ 6015.043658] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6015.051731] LustreError: 199587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6015.336150] Lustre: server umount lustre-MDT0001 complete [ 6025.234652] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6028.154960] Key type lgssc unregistered [ 6028.413866] LNet: 201628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6028.421420] LNetError: 201628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6028.437877] LNet: Removed LNI 192.168.203.157@tcp [ 6029.180761] Key type .llcrypt unregistered [ 6029.184682] Key type ._llcrypt unregistered [ 6047.253119] Key type ._llcrypt registered [ 6047.257334] Key type .llcrypt registered [ 6047.335861] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6048.605226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6048.791473] alg: No test for adler32 (adler32-zlib) [ 6049.997442] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6050.278927] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6051.983248] Key type lgssc registered [ 6053.155172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6061.615244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6061.648500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.174878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6068.528395] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6073.318719] LustreError: 202897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.369140] LustreError: 202897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6077.798641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6078.323143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6082.876894] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6092.321214] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6099.538800] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6106.882269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6107.161454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.228474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6113.913761] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6122.581532] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6128.298509] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6137.780199] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6149.663444] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6155.098418] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6164.455016] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.467108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6164.476843] Lustre: Skipped 1 previous similar message [ 6164.486788] Lustre: Skipped 1 previous similar message [ 6168.424660] Lustre: server umount lustre-OST0000 complete [ 6174.696365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.699275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6174.712779] Lustre: Skipped 1 previous similar message [ 6174.728384] Lustre: Skipped 1 previous similar message [ 6178.377310] Lustre: server umount lustre-MDT0000 complete [ 6179.809607] LustreError: 202892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.823183] LustreError: 202892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6182.203489] LustreError: 202879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633251 with bad export cookie 15296979061295557233 [ 6182.205494] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6182.217319] LustreError: 202879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6182.762291] Lustre: server umount lustre-MDT0001 complete [ 6194.815117] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6197.366725] Key type lgssc unregistered [ 6197.632209] LNet: 207785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6197.646826] LNetError: 207785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6197.666849] LNet: Removed LNI 192.168.203.157@tcp [ 6198.474115] Key type .llcrypt unregistered [ 6198.476329] Key type ._llcrypt unregistered [ 6222.625423] Key type ._llcrypt registered [ 6222.634160] Key type .llcrypt registered [ 6222.756769] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6223.731662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6223.844155] alg: No test for adler32 (adler32-zlib) [ 6224.924678] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6225.149892] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6226.863160] Key type lgssc registered [ 6227.801818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6235.995651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6236.007975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.432423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6241.813203] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6247.906789] LustreError: 209200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.942348] LustreError: 209200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6250.008127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6250.298624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6254.614837] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6263.417503] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6270.462813] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6277.885292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6278.443555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6282.568442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6285.299599] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6293.578776] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6311.392870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6311.402496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.416573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6311.418401] Lustre: Skipped 1 previous similar message [ 6311.434020] Lustre: Skipped 1 previous similar message [ 6315.843766] Lustre: server umount lustre-OST0000 complete [ 6321.637714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6321.661878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6321.666724] Lustre: Skipped 1 previous similar message [ 6326.511933] Lustre: server umount lustre-MDT0000 complete [ 6326.755488] LustreError: 209201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6326.770813] LustreError: 209201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6330.516811] LustreError: 209180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633399 with bad export cookie 13526164825950076463 [ 6330.525067] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6330.528447] LustreError: 209180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6330.928583] Lustre: server umount lustre-MDT0001 complete [ 6345.145490] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6348.251420] Key type lgssc unregistered [ 6348.532682] LNet: 212889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6348.536302] LNetError: 212889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6348.563592] LNet: Removed LNI 192.168.203.157@tcp [ 6349.308431] Key type .llcrypt unregistered [ 6349.312496] Key type ._llcrypt unregistered [ 6372.176248] Key type ._llcrypt registered [ 6372.181794] Key type .llcrypt registered [ 6372.281066] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6373.585334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6373.612233] alg: No test for adler32 (adler32-zlib) [ 6374.829236] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6375.234262] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6376.919146] Key type lgssc registered [ 6378.134987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6385.340624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6385.363813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6386.756385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6391.057681] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6396.899221] LustreError: 214156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6396.925359] LustreError: 214156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6399.879702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.263228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6404.909597] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6413.154930] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6419.609669] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6426.591909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6426.961341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6431.089133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6434.132340] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6445.529740] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6466.021903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6466.030434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6466.040589] Lustre: Skipped 1 previous similar message [ 6466.061177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6468.675925] Lustre: server umount lustre-OST0000 complete [ 6476.258079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6476.290731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6476.297878] Lustre: Skipped 2 previous similar messages [ 6479.209774] Lustre: server umount lustre-MDT0000 complete [ 6481.378699] LustreError: 214884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.409170] LustreError: 214884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6482.761084] LustreError: 217088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633551 with bad export cookie 442563117499401626 [ 6482.771795] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6482.781148] LustreError: 217088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6483.140463] Lustre: server umount lustre-MDT0001 complete [ 6494.072234] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6496.871711] Key type lgssc unregistered [ 6497.117410] LNet: 217843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6497.127759] LNetError: 217843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6497.142935] LNet: Removed LNI 192.168.203.157@tcp [ 6498.090185] Key type .llcrypt unregistered [ 6498.092450] Key type ._llcrypt unregistered [ 6517.184090] Key type ._llcrypt registered [ 6517.191891] Key type .llcrypt registered [ 6517.294298] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6518.245263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6518.268188] alg: No test for adler32 (adler32-zlib) [ 6519.323445] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6519.562514] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6521.247178] Key type lgssc registered [ 6522.272917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6529.103399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6529.130429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6530.643165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6534.880554] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6540.778173] LustreError: 219110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.789499] LustreError: 219110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6543.289809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6543.635940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6547.881395] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6555.271567] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6560.122236] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6566.181813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6566.496395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6572.921915] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6575.733649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6581.468516] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6590.209756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6590.272855] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6590.513969] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6596.796141] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6604.460953] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6606.065812] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6617.782984] Lustre: Permanently deactivating lustre-OST0001 [ 6617.815252] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6626.865640] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6632.479120] Lustre: Permanently reactivating lustre-OST0001 [ 6632.501690] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6632.508451] Lustre: Skipped 2 previous similar messages [ 6634.345428] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6634.361068] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6634.367497] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6634.378391] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6634.402065] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6643.685728] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6643.701361] Lustre: Skipped 1 previous similar message [ 6643.710820] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6643.715575] Lustre: Skipped 1 previous similar message [ 6645.468600] Lustre: server umount lustre-OST0001 complete [ 6648.816791] LustreError: 220931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6648.835867] LustreError: 220931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6650.849534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6650.861291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.878027] Lustre: Skipped 1 previous similar message [ 6650.883368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6652.896566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6652.909454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6652.920447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6653.928441] LustreError: 222666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6653.940688] LustreError: 222666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6655.658743] Lustre: server umount lustre-OST0000 complete [ 6661.602911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6661.609920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6661.628651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6661.640496] Lustre: Skipped 2 previous similar messages [ 6665.648868] Lustre: server umount lustre-MDT0000 complete [ 6668.749910] LustreError: 219091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786633737 with bad export cookie 8160385240545918291 [ 6668.755826] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6668.765642] LustreError: 219091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6669.050630] Lustre: server umount lustre-MDT0001 complete [ 6682.925880] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 6686.550696] Key type lgssc unregistered [ 6686.928980] LNet: 224156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6686.941181] LNetError: 224156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6686.974591] LNet: Removed LNI 192.168.203.157@tcp [ 6688.559879] Key type .llcrypt unregistered [ 6688.567832] Key type ._llcrypt unregistered [ 6730.961426] Key type ._llcrypt registered [ 6730.964520] Key type .llcrypt registered [ 6731.052752] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 6731.835380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6731.888827] alg: No test for adler32 (adler32-zlib) [ 6732.924712] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 6733.148593] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 6734.815344] Key type lgssc registered [ 6736.157405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6744.721960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6744.750239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.229709] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6746.278518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6746.495889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6750.873371] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6759.884627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.955289] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6759.985800] Lustre: 226362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6760.185568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6764.345306] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6771.927416] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6777.581324] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6784.584296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6784.837666] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6785.069416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6791.200489] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 6794.881551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6799.238413] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6802.077422] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6806.449642] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6816.469859] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6821.908599] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6830.400360] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6850.509172] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6850.513440] Lustre: Skipped 2 previous similar messages [ 6884.630619] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6884.637073] Lustre: Skipped 3 previous similar messages [ 6955.704161] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6955.719459] Lustre: Skipped 8 previous similar messages [ 6979.041632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6979.063813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6979.099071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6979.556593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6980.829318] Lustre: server umount lustre-OST0000 complete [ 6985.697494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6985.718409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6985.739268] Lustre: Skipped 1 previous similar message [ 6985.747681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6985.761632] Lustre: Skipped 1 previous similar message [ 6989.792476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6989.792553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.296517] Lustre: server umount lustre-MDT0000 complete [ 6995.256317] LustreError: 229315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634064 with bad export cookie 17080294198958774106 [ 6995.263235] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6995.272961] LustreError: 229315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6995.688189] Lustre: server umount lustre-MDT0001 complete [ 7007.506687] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7010.838142] Key type lgssc unregistered [ 7011.299481] LNet: 231159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7011.316666] LNetError: 231159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7011.337601] LNet: Removed LNI 192.168.203.157@tcp [ 7012.275157] Key type .llcrypt unregistered [ 7012.278602] Key type ._llcrypt unregistered [ 7037.899771] Key type ._llcrypt registered [ 7037.904225] Key type .llcrypt registered [ 7038.001541] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7039.437331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7039.555243] alg: No test for adler32 (adler32-zlib) [ 7040.603980] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7040.849979] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7042.519779] Key type lgssc registered [ 7043.846421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7051.812423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7051.858153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.394933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7058.429091] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7063.523864] LustreError: 232576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7063.546344] LustreError: 232576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7067.076026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7067.387101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7071.849150] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7079.849355] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7086.305532] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7093.911429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7094.350138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7098.474232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7100.660900] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_default_debug -1 all [ 7107.736449] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7124.449031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7124.454919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.471294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7125.543814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7125.550759] Lustre: Skipped 2 previous similar messages [ 7126.262985] Lustre: server umount lustre-OST0000 complete [ 7130.594598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7130.598690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7130.610558] Lustre: Skipped 2 previous similar messages [ 7130.626568] Lustre: Skipped 1 previous similar message [ 7136.551824] Lustre: server umount lustre-MDT0000 complete [ 7136.741065] LustreError: 232576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7136.749030] LustreError: 232576:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7139.878301] LustreError: 235156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786634208 with bad export cookie 13226115186159541495 [ 7139.889638] LustreError: MGC192.168.203.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7140.516904] Lustre: server umount lustre-MDT0001 complete [ 7153.876714] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7157.661640] Key type lgssc unregistered [ 7158.029070] LNet: 236276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7158.050046] LNetError: 236276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7158.073756] LNet: Removed LNI 192.168.203.157@tcp [ 7159.053905] Key type .llcrypt unregistered [ 7159.056165] Key type ._llcrypt unregistered [ 7183.258778] Key type ._llcrypt registered [ 7183.264031] Key type .llcrypt registered [ 7183.388727] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7184.344671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7184.441902] alg: No test for adler32 (adler32-zlib) [ 7185.524673] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7185.784346] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7187.465511] Key type lgssc registered [ 7188.552577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7457.064984] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7459.454307] Key type lgssc unregistered [ 7459.789262] LNet: 238305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7459.801378] LNetError: 238305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7459.827953] LNet: Removed LNI 192.168.203.157@tcp [ 7460.450329] Key type .llcrypt unregistered [ 7460.452532] Key type ._llcrypt unregistered [ 7482.115559] Key type ._llcrypt registered [ 7482.117024] Key type .llcrypt registered [ 7482.236849] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing set_hostid [ 7498.219408] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing load_modules_local [ 7499.275475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7499.371032] alg: No test for adler32 (adler32-zlib) [ 7500.533907] Lustre: Lustre: Build Version: 2.17.57_1_gb6c95b3 [ 7500.721929] LNet: Added LNI 192.168.203.157@tcp [8/256/0/180] [ 7502.359159] Key type lgssc registered [ 7503.414384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7534.112236] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7296 sec ========== 11:23:21 (1786634601) [ 7536.100289] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:23:23 (1786634603) === [ 7539.559761] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:23:27 (1786634607) === [ 7576.055937] Lustre: DEBUG MARKER: oleg357-server.virtnet: executing unload_modules_local [ 7578.802219] Key type lgssc unregistered [ 7579.109324] LNet: 243545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7579.113498] LNetError: 243545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7579.129767] LNet: Removed LNI 192.168.203.157@tcp [ 7580.000651] Key type .llcrypt unregistered [ 7580.004593] Key type ._llcrypt unregistered