[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 573666901 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003123] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011120] LSM: Security Framework initializing [ 0.012041] Yama: becoming mindful. [ 0.013038] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.021493] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026267] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028024] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030101] *** VALIDATE tmpfs *** [ 0.031432] *** VALIDATE proc *** [ 0.033225] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035256] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036143] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038023] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.041443] debug: unmapping init [mem 0xffffffffa2259000-0xffffffffa2260fff] [ 0.043220] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045019] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050010] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052259] rcu: Hierarchical SRCU implementation. [ 0.054344] smp: Bringing up secondary CPUs ... [ 0.055556] x86: Booting SMP configuration: [ 0.056019] .... node #0, CPUs: #1 #2 #3 [ 0.059288] smp: Brought up 1 node, 4 CPUs [ 0.061008] smpboot: Max logical packages: 1 [ 0.062012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126042] node 0 deferred pages initialised in 62ms [ 0.128527] devtmpfs: initialized [ 0.129217] x86/mm: Memory block size: 128MB [ 0.132856] gcov: version magic: 0x41383552 [ 0.135270] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136065] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137286] pinctrl core: initialized pinctrl subsystem [ 0.139190] [ 0.139984] ************************************************************* [ 0.143012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146010] ** ** [ 0.149011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151008] ** ** [ 0.154011] ** This means that this kernel is built to expose internal ** [ 0.157012] ** IOMMU data structures, which may compromise security on ** [ 0.159011] ** your system. ** [ 0.162011] ** ** [ 0.165011] ** If you see this message and you are not debugging the ** [ 0.168016] ** kernel, report this immediately to your vendor! ** [ 0.170010] ** ** [ 0.173008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174007] ************************************************************* [ 0.178211] NET: Registered protocol family 16 [ 0.179493] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.188031] cpuidle: using governor menu [ 0.189722] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.192434] PCI: Using configuration type 1 for base access [ 0.195112] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205042] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206057] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.208106] cryptd: max_cpu_qlen set to 1000 [ 0.210156] ACPI: Added _OSI(Module Device) [ 0.211008] ACPI: Added _OSI(Processor Device) [ 0.212008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218579] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224397] ACPI: Interpreter enabled [ 0.226043] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227018] ACPI: Using IOAPIC for interrupt routing [ 0.228076] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231336] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.242600] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.247015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.250068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.255212] acpiphp: Slot [2] registered [ 0.257102] acpiphp: Slot [5] registered [ 0.259091] acpiphp: Slot [6] registered [ 0.261085] acpiphp: Slot [7] registered [ 0.263106] acpiphp: Slot [8] registered [ 0.264083] acpiphp: Slot [9] registered [ 0.266099] acpiphp: Slot [10] registered [ 0.268095] acpiphp: Slot [3] registered [ 0.269148] acpiphp: Slot [4] registered [ 0.271086] acpiphp: Slot [11] registered [ 0.273080] acpiphp: Slot [12] registered [ 0.274071] acpiphp: Slot [13] registered [ 0.276092] acpiphp: Slot [14] registered [ 0.278067] acpiphp: Slot [15] registered [ 0.279000] acpiphp: Slot [16] registered [ 0.280069] acpiphp: Slot [17] registered [ 0.281070] acpiphp: Slot [18] registered [ 0.282069] acpiphp: Slot [19] registered [ 0.284065] acpiphp: Slot [20] registered [ 0.285064] acpiphp: Slot [21] registered [ 0.286068] acpiphp: Slot [22] registered [ 0.288081] acpiphp: Slot [23] registered [ 0.289060] acpiphp: Slot [24] registered [ 0.290059] acpiphp: Slot [25] registered [ 0.291060] acpiphp: Slot [26] registered [ 0.293092] acpiphp: Slot [27] registered [ 0.294114] acpiphp: Slot [28] registered [ 0.295098] acpiphp: Slot [29] registered [ 0.297062] acpiphp: Slot [30] registered [ 0.298065] acpiphp: Slot [31] registered [ 0.300051] PCI host bridge to bus 0000:00 [ 0.301013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.303026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.306020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.309021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.311015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.314019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.315168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.318025] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.321233] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.330724] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.336051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.344013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346553] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.351923] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354029] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356816] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.376011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.381011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.386251] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.393017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.399013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422998] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.433013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.452012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.516012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.528210] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.539013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.552013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.614012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.639505] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.654014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.666014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.705026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.727390] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.753021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.777013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.824014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.839491] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.854013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.868013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.897013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.912040] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.914298] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.916462] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.919674] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.922184] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.927076] iommu: Default domain type: Passthrough [ 0.928331] SCSI subsystem initialized [ 0.930122] ACPI: bus type USB registered [ 0.931070] usbcore: registered new interface driver usbfs [ 0.933052] usbcore: registered new interface driver hub [ 0.935064] usbcore: registered new device driver usb [ 0.937140] pps_core: LinuxPPS API ver. 1 registered [ 0.938008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.941049] PTP clock support registered [ 0.943117] EDAC MC: Ver: 3.0.0 [ 0.945105] PCI: Using ACPI for IRQ routing [ 0.946848] NetLabel: Initializing [ 0.948007] NetLabel: domain hash size = 128 [ 0.949006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.951064] NetLabel: unlabeled traffic allowed by default [ 0.953081] vgaarb: loaded [ 0.955064] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.956007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.964329] clocksource: Switched to clocksource kvm-clock [ 1.070458] VFS: Disk quotas dquot_6.6.0 [ 1.072879] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.075079] *** VALIDATE ramfs *** [ 1.076046] *** VALIDATE hugetlbfs *** [ 1.077262] pnp: PnP ACPI init [ 1.079300] pnp: PnP ACPI: found 6 devices [ 1.127607] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.130842] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.133302] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.135668] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.138143] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.140440] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.143815] NET: Registered protocol family 2 [ 1.145707] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.149481] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.152142] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.156878] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.159731] TCP: Hash tables configured (established 65536 bind 65536) [ 1.161926] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.164125] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.166154] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.168339] NET: Registered protocol family 1 [ 1.170586] RPC: Registered named UNIX socket transport module. [ 1.172581] RPC: Registered udp transport module. [ 1.173878] RPC: Registered tcp transport module. [ 1.175457] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.177777] NET: Registered protocol family 44 [ 1.179455] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.181941] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.184330] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.186906] PCI: CLS 0 bytes, default 64 [ 1.189340] Unpacking initramfs... [ 2.602382] debug: unmapping init [mem 0xffff966c7cc54000-0xffff966c7ffbffff] [ 2.606278] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.608760] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.612165] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.118639] Initialise system trusted keyrings [ 3.120719] Key type blacklist registered [ 3.122743] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.132213] zbud: loaded [ 3.135410] *** VALIDATE nfs *** [ 3.136863] *** VALIDATE nfs4 *** [ 3.138536] pstore: using deflate compression [ 3.141964] Platform Keyring initialized [ 3.238073] NET: Registered protocol family 38 [ 3.240086] Key type asymmetric registered [ 3.241569] Asymmetric key parser 'x509' registered [ 3.243751] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.246713] io scheduler mq-deadline registered [ 3.248202] io scheduler kyber registered [ 3.249823] io scheduler bfq registered [ 3.252050] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.254862] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.257574] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.260568] ACPI: Power Button [PWRF] [ 3.266190] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.272550] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.308189] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.317618] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.338777] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.365900] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.395352] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.401036] Non-volatile memory driver v1.3 [ 3.402696] Linux agpgart interface v0.103 [ 3.433028] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.436563] vda: detected capacity change from 0 to 74686464 [ 3.462640] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.465813] vdb: detected capacity change from 0 to 1073741824 [ 3.491665] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.494791] vdc: detected capacity change from 0 to 2621440000 [ 3.513812] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.517183] vdd: detected capacity change from 0 to 2621440000 [ 3.532030] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.535388] vde: detected capacity change from 0 to 4294967296 [ 3.560227] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.564217] vdf: detected capacity change from 0 to 4294967296 [ 3.575220] libphy: Fixed MDIO Bus: probed [ 3.580465] usbcore: registered new interface driver usbserial_generic [ 3.583085] usbserial: USB Serial support registered for generic [ 3.585543] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.590614] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.592401] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.594516] mousedev: PS/2 mouse device common for all mice [ 3.597384] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.599752] rtc_cmos 00:05: RTC can wake from S4 [ 3.605314] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.605465] rtc_cmos 00:05: registered as rtc0 [ 3.611264] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.614441] intel_pstate: CPU model not supported [ 3.616714] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.621661] hid: raw HID events driver (C) Jiri Kosina [ 3.624094] usbcore: registered new interface driver usbhid [ 3.626135] usbhid: USB HID core driver [ 3.627374] drop_monitor: Initializing network drop monitor service [ 3.629259] Initializing XFRM netlink socket [ 3.630797] NET: Registered protocol family 10 [ 3.633756] Segment Routing with IPv6 [ 3.634970] NET: Registered protocol family 17 [ 3.637000] mpls_gso: MPLS GSO support [ 3.642680] RAS: Correctable Errors collector initialized. [ 3.644898] AVX version of gcm_enc/dec engaged. [ 3.646612] AES CTR mode by8 optimization enabled [ 3.722537] sched_clock: Marking stable (3722507090, 0)->(4613671240, -891164150) [ 3.726646] registered taskstats version 1 [ 3.729275] Loading compiled-in X.509 certificates [ 3.731295] zswap: loaded using pool lzo/zbud [ 3.755179] Key type big_key registered [ 3.768143] Key type encrypted registered [ 3.769839] ima: No TPM chip found, activating TPM-bypass! [ 3.771710] ima: Allocated hash algorithm: sha1 [ 3.773224] ima: No architecture policies found [ 3.774800] evm: Initialising EVM extended attributes: [ 3.776587] evm: security.selinux [ 3.777875] evm: security.ima [ 3.778922] evm: security.capability [ 3.780174] evm: HMAC attrs: 0x1 [ 3.782619] rtc_cmos 00:05: setting system clock to 2026-08-03 19:41:51 UTC (1785786111) [ 3.788721] debug: unmapping init [mem 0xffffffffa3203000-0xffffffffa33fffff] [ 3.792091] debug: unmapping init [mem 0xffffffffa1f82000-0xffffffffa2258fff] [ 3.800064] Write protecting the kernel read-only data: 28672k [ 3.803090] debug: unmapping init [mem 0xffffffffa0603000-0xffffffffa07fffff] [ 3.806016] debug: unmapping init [mem 0xffffffffa0f14000-0xffffffffa0ffffff] [ 3.838250] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.845702] systemd[1]: Detected virtualization kvm. [ 3.847168] systemd[1]: Detected architecture x86-64. [ 3.848851] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.877188] systemd[1]: No hostname configured. [ 3.878927] systemd[1]: Set hostname to . [ 3.880932] random: systemd: uninitialized urandom read (16 bytes read) [ 3.883151] systemd[1]: Initializing machine ID from random generator. [ 3.919980] random: ln: uninitialized urandom read (6 bytes read) [ 4.027773] random: systemd: uninitialized urandom read (16 bytes read) [ 4.030180] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.036299] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.040311] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.604475] device-mapper: uevent: version 1.0.3 [ 4.606339] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.330476] virtio_net virtio0 ens2: renamed from eth0 [ 5.335697] random: fast init done [ 5.410211] scsi host0: ata_piix [ 5.424896] scsi host1: ata_piix [ 5.426296] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.428353] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.792346] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.289322] random: crng init done [ 10.290911] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.728764] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.862286] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.132294] SELinux: Disabled at runtime. [ 12.192289] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.202832] systemd[1]: Detected virtualization kvm. [ 12.205101] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.687193] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.691133] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.697520] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.703326] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.706480] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.713715] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.722977] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.829448] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.202212] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.524726] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.588190] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.745626] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.763372] EDAC sbridge: Ver: 1.1.2 [ 15.260836] Key type dns_resolver registered [ 15.580387] NFS: Registering the id_resolver key type [ 15.583527] Key type id_resolver registered [ 15.585430] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg359-server login: [ 53.304194] hrtimer: interrupt took 22909575 ns [ 104.315418] libcfs: loading out-of-tree module taints kernel. [ 104.452151] Key type ._llcrypt registered [ 104.461274] Key type .llcrypt registered [ 104.703621] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 134.451347] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 136.649962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 136.669551] alg: No test for adler32 (adler32-zlib) [ 138.300591] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 139.930566] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 141.704792] Key type lgssc registered [ 145.172704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 169.341384] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 234.457839] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 251.651328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 251.680928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 253.029364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 253.071342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 253.198927] Lustre: lustre-MDT0000: new disk, initializing [ 253.362383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.399527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 259.218888] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 278.122078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 278.296350] Lustre: 6538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 278.391295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 278.402412] Lustre: Skipped 1 previous similar message [ 278.617545] Lustre: lustre-MDT0001: new disk, initializing [ 278.732422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.772647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 278.791942] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 284.070301] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 289.732689] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 303.659294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 304.228597] Lustre: lustre-OST0000: new disk, initializing [ 304.234457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 304.246568] Lustre: 8480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 304.341216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 312.988292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 313.013659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 313.292116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 314.994735] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 334.345821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 334.521377] Lustre: lustre-OST0001: new disk, initializing [ 334.525874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 334.536138] Lustre: 9552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 334.631679] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 341.548114] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 342.153874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 342.175860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 342.281537] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 356.428563] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 365.243416] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 374.395245] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing check_logdir /tmp/testlogs/ [ 381.494899] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing yml_node [ 385.790092] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 389.268831] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 392.845759] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 394.792393] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 3 15:48:19 EDT 2026 [ 418.937181] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 421.204475] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 439.776558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 439.779167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 439.796192] Lustre: Skipped 1 previous similar message [ 444.897397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 444.900114] Lustre: Skipped 6 previous similar messages [ 445.757418] Lustre: server umount lustre-MDT0000 complete [ 450.252586] LustreError: 6532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785786557 with bad export cookie 12344368188557995436 [ 450.256565] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.270268] LustreError: 6532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 455.137772] LustreError: 6547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.144971] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.168133] LustreError: 6547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 455.205667] Lustre: Skipped 2 previous similar messages [ 455.209306] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 455.217175] Lustre: Skipped 1 previous similar message [ 456.895438] Lustre: server umount lustre-MDT0001 complete [ 461.616917] Lustre: server umount lustre-OST0000 complete [ 466.507683] Lustre: server umount lustre-OST0001 complete [ 474.156559] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 485.997058] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 533.324531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.881739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 533.943818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 534.107819] Lustre: lustre-MDT0000: new disk, initializing [ 534.242732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 534.279930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 541.806342] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 556.358849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.460701] Lustre: 15109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 556.548212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 556.551265] Lustre: Skipped 1 previous similar message [ 556.616836] Lustre: lustre-MDT0001: new disk, initializing [ 556.677178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 556.698150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 556.708151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 562.609296] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 575.548817] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 582.344460] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 590.378701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 590.953359] Lustre: lustre-OST0000: new disk, initializing [ 590.966437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 590.979713] Lustre: 17077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 591.108327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 592.410762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 592.423057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 592.611055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 598.956727] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 610.431680] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 616.119370] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 616.364407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 622.428742] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 622.731313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 628.711605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.738021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.748764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 628.763167] Lustre: Skipped 2 previous similar messages [ 628.786329] Lustre: Skipped 1 previous similar message [ 631.270935] Lustre: server umount lustre-OST0000 complete [ 638.944406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.949793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 638.974797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.984429] LustreError: Skipped 1 previous similar message [ 639.003368] Lustre: Skipped 1 previous similar message [ 649.187521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.204259] Lustre: Skipped 3 previous similar messages [ 651.232847] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 651.728872] Lustre: server umount lustre-MDT0000 complete [ 654.304674] LustreError: 15120:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 654.339118] LustreError: 15120:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 658.794524] LustreError: 15101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785786766 with bad export cookie 12344368188557996969 [ 658.809572] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.843924] LustreError: 15101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 660.723840] Lustre: server umount lustre-MDT0001 complete [ 667.591111] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 15:52:50 (1785786770) [ 678.453398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.407424] LustreError: 19347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.527038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 684.534069] LustreError: 19348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.177937] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 689.637657] LustreError: 19347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.450256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 703.880094] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 713.207790] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 721.034424] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 730.640388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 739.036596] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 748.113682] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 766.435122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.441525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 766.465585] Lustre: Skipped 2 previous similar messages [ 766.491524] Lustre: Skipped 1 previous similar message [ 771.563956] LustreError: 21853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 771.591842] LustreError: 21853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 771.784160] Lustre: server umount lustre-OST0000 complete [ 779.745596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 779.758864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.227479] Lustre: server umount lustre-MDT0000 complete [ 789.458222] LustreError: 19329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785786897 with bad export cookie 12344368188557997851 [ 789.465867] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 789.482127] LustreError: 19329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 789.913986] Lustre: server umount lustre-MDT0001 complete [ 805.658901] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 809.122737] Key type lgssc unregistered [ 809.526568] LNet: 22908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.543884] LNetError: 22908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.598443] LNet: Removed LNI 192.168.203.159@tcp [ 811.987119] Key type .llcrypt unregistered [ 811.988700] Key type ._llcrypt unregistered [ 838.721306] Key type ._llcrypt registered [ 838.724392] Key type .llcrypt registered [ 838.894129] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 840.327915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 840.479352] alg: No test for adler32 (adler32-zlib) [ 841.861062] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 842.227882] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 844.034552] Key type lgssc registered [ 845.685698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 855.581583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 855.613130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.214124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.971093] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 866.792238] LustreError: 24323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 866.814525] LustreError: 24323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 871.905340] LustreError: 24324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.731983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.391177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 880.935349] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 891.186687] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 898.639634] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 907.198276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.764177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 915.967661] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 925.776894] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 927.238818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 942.568931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 942.571329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.584330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 942.616875] Lustre: Skipped 1 previous similar message [ 945.294223] Lustre: server umount lustre-OST0000 complete [ 952.801427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.835750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 952.889456] Lustre: Skipped 2 previous similar messages [ 957.963702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 959.992256] Lustre: server umount lustre-MDT0000 complete [ 963.040265] LustreError: 25053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 963.090232] LustreError: 25053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 969.202069] LustreError: 25053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.238037] LustreError: 25053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 972.532483] LustreError: 24305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787080 with bad export cookie 14210084127531224656 [ 972.591391] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.374909] Lustre: server umount lustre-MDT0001 complete [ 1003.061270] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1007.015403] Key type lgssc unregistered [ 1007.507704] LNet: 28329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.513601] LNetError: 28329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.530622] LNet: Removed LNI 192.168.203.159@tcp [ 1008.858933] Key type .llcrypt unregistered [ 1008.863304] Key type ._llcrypt unregistered [ 1039.277595] Key type ._llcrypt registered [ 1039.279674] Key type .llcrypt registered [ 1039.374244] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1041.064785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1041.118947] alg: No test for adler32 (adler32-zlib) [ 1042.219798] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1042.469224] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1044.119476] Key type lgssc registered [ 1045.377938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1056.123484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1056.151704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.715395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.088890] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1068.006269] LustreError: 29742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1068.028467] LustreError: 29742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1073.121488] LustreError: 29741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1074.122558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.596717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1081.960785] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1092.305619] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1101.677864] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1120.044413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1120.628768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1127.799475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1128.270647] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1140.397153] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1155.556781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.571768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1155.596028] Lustre: Skipped 1 previous similar message [ 1155.613399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1155.623795] Lustre: Skipped 1 previous similar message [ 1160.686335] LustreError: 31908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.720212] LustreError: 31908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1161.469644] Lustre: server umount lustre-OST0000 complete [ 1170.915078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.937103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1170.955771] Lustre: Skipped 1 previous similar message [ 1176.041675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1181.151825] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1181.171748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1181.186859] Lustre: Skipped 2 previous similar messages [ 1181.437046] Lustre: server umount lustre-MDT0000 complete [ 1186.241446] LustreError: 29723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787293 with bad export cookie 10256903896629739952 [ 1186.273872] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1187.146944] Lustre: server umount lustre-MDT0001 complete [ 1203.587914] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1207.237505] Key type lgssc unregistered [ 1207.543688] LNet: 33657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.567660] LNetError: 33657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.636169] LNet: Removed LNI 192.168.203.159@tcp [ 1209.864444] Key type .llcrypt unregistered [ 1209.866196] Key type ._llcrypt unregistered [ 1241.237073] Key type ._llcrypt registered [ 1241.239666] Key type .llcrypt registered [ 1241.342969] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1242.764474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1242.837547] alg: No test for adler32 (adler32-zlib) [ 1244.217597] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1244.559235] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1246.343881] Key type lgssc registered [ 1247.993970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1257.432552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1257.442937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.990271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.286187] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1269.218030] LustreError: 35072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.243130] LustreError: 35072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1273.853171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1274.301882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1279.737456] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1289.401548] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1298.048710] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1308.675862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1309.296758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.733903] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1328.774904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1328.894075] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1344.480237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1344.485058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1344.495729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1344.511375] Lustre: Skipped 1 previous similar message [ 1344.528812] Lustre: Skipped 1 previous similar message [ 1347.590728] Lustre: server umount lustre-OST0000 complete [ 1354.721659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.724366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1354.741232] Lustre: Skipped 1 previous similar message [ 1358.602036] Lustre: server umount lustre-MDT0000 complete [ 1359.843119] LustreError: 35073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.872684] LustreError: 35073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1363.126813] LustreError: 35052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787470 with bad export cookie 5431504703125742962 [ 1363.139798] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1363.144617] LustreError: 35052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1364.204724] Lustre: server umount lustre-MDT0001 complete [ 1382.578321] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1385.462323] Key type lgssc unregistered [ 1385.712090] LNet: 38635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.716028] LNetError: 38635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.749176] LNet: Removed LNI 192.168.203.159@tcp [ 1386.876054] Key type .llcrypt unregistered [ 1386.879221] Key type ._llcrypt unregistered [ 1424.504320] Key type ._llcrypt registered [ 1424.509256] Key type .llcrypt registered [ 1424.654371] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1426.508688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1426.544889] alg: No test for adler32 (adler32-zlib) [ 1427.837632] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1428.306601] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1430.031141] Key type lgssc registered [ 1431.510679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1441.406182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1441.426650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.033209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.677357] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1452.524277] LustreError: 40053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.550854] LustreError: 40053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1457.640815] LustreError: 40052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.756297] LustreError: 40053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.052312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.536630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1470.129760] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1480.776882] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1488.438211] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1496.696682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1497.139765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1503.723735] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1512.496302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1513.288067] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1521.373566] Lustre: lustre-OST0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 1522.665827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.668421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1522.679415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1522.685940] Lustre: Skipped 1 previous similar message [ 1526.445532] Lustre: lustre-OST0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 1526.461412] Lustre: Skipped 1 previous similar message [ 1527.698150] Lustre: server umount lustre-OST0000 complete [ 1553.378717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1553.397515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1553.403901] Lustre: Skipped 1 previous similar message [ 1558.497742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.176370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1566.535716] Lustre: server umount lustre-MDT0000 complete [ 1567.718250] LustreError: 40048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1567.744425] LustreError: 40048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1571.349297] LustreError: 40035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787679 with bad export cookie 14045907239636250577 [ 1571.362772] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1578.024268] Lustre: server umount lustre-MDT0001 complete [ 1593.701100] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1597.526388] Key type lgssc unregistered [ 1597.947588] LNet: 43765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.962466] LNetError: 43765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.985141] LNet: Removed LNI 192.168.203.159@tcp [ 1599.293469] Key type .llcrypt unregistered [ 1599.296903] Key type ._llcrypt unregistered [ 1626.997741] Key type ._llcrypt registered [ 1627.009289] Key type .llcrypt registered [ 1627.128205] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1628.335192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1628.921037] alg: No test for adler32 (adler32-zlib) [ 1630.351280] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1630.684204] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1632.407111] Key type lgssc registered [ 1634.241096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1644.588186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1644.622390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.211481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.498223] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1655.780621] LustreError: 45176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.793941] LustreError: 45176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1660.897141] LustreError: 45177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1661.375982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1662.096512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1668.642566] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1680.611363] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1690.156391] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1700.386517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1700.907214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.685585] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1716.276200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1718.093162] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1726.168705] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 1726.434028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.449747] Lustre: Skipped 1 previous similar message [ 1731.254507] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 1731.281798] Lustre: Skipped 3 previous similar messages [ 1732.645972] Lustre: server umount lustre-MDT0000 complete [ 1736.353039] LustreError: 45173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1738.722957] LustreError: 45159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785787846 with bad export cookie 4474268207261282121 [ 1738.731768] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1738.745030] LustreError: 45159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1741.468768] LustreError: 45171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.516895] LustreError: 45171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1741.543840] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.59@tcp (stopping) [ 1741.554501] Lustre: Skipped 3 previous similar messages [ 1741.814351] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.866789] Lustre: Skipped 1 previous similar message [ 1743.846609] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1743.858614] Lustre: Skipped 1 previous similar message [ 1745.267102] Lustre: server umount lustre-MDT0001 complete [ 1762.237789] Lustre: server umount lustre-OST0000 complete [ 1784.689698] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 1788.258592] Key type lgssc unregistered [ 1788.642834] LNet: 49040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.654145] LNetError: 49040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.706298] LNet: Removed LNI 192.168.203.159@tcp [ 1789.869123] Key type .llcrypt unregistered [ 1789.870860] Key type ._llcrypt unregistered [ 1819.617193] Key type ._llcrypt registered [ 1819.622952] Key type .llcrypt registered [ 1819.796425] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 1821.049451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1821.102920] alg: No test for adler32 (adler32-zlib) [ 1822.407631] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 1822.724584] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 1824.431190] Key type lgssc registered [ 1825.655336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1835.608061] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1835.652052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1851.297804] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1856.417548] LustreError: 50445:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 1884.447329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.885176] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 1904.002476] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2070.674762] Lustre: server umount lustre-OST0000 complete [ 2094.645257] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2098.101658] Key type lgssc unregistered [ 2098.588145] LNet: 52127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.601869] LNetError: 52127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2099.623168] LNet: Removed LNI 192.168.203.159@tcp [ 2100.930498] Key type .llcrypt unregistered [ 2100.938427] Key type ._llcrypt unregistered [ 2134.335470] Key type ._llcrypt registered [ 2134.337384] Key type .llcrypt registered [ 2134.416079] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2135.830478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2135.872651] alg: No test for adler32 (adler32-zlib) [ 2137.126215] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2137.489840] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2139.247279] Key type lgssc registered [ 2140.470476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2150.964794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2151.000452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.778667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.787622] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2163.170044] LustreError: 53546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.202578] LustreError: 53546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2168.321939] LustreError: 53545:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.729976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.468177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2175.632200] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2184.609181] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2191.105232] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2200.692731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2201.267978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.629988] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2218.876892] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2222.692811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.694778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2222.727480] Lustre: Skipped 1 previous similar message [ 2228.185107] Lustre: server umount lustre-OST0000 complete [ 2233.312292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.320075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2233.336081] Lustre: Skipped 1 previous similar message [ 2238.445990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2238.461912] Lustre: Skipped 2 previous similar messages [ 2239.727536] Lustre: server umount lustre-MDT0000 complete [ 2242.530624] LustreError: 54273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.554235] LustreError: 54273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2245.216901] LustreError: 53526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788352 with bad export cookie 13592964284031173191 [ 2245.226156] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2245.234479] LustreError: 53526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2245.945716] Lustre: server umount lustre-MDT0001 complete [ 2261.260660] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2264.506757] Key type lgssc unregistered [ 2264.922644] LNet: 57080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.934022] LNetError: 57080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2264.958365] LNet: Removed LNI 192.168.203.159@tcp [ 2266.241651] Key type .llcrypt unregistered [ 2266.243541] Key type ._llcrypt unregistered [ 2296.326920] Key type ._llcrypt registered [ 2296.332895] Key type .llcrypt registered [ 2296.495381] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2298.198860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2298.325178] alg: No test for adler32 (adler32-zlib) [ 2299.606947] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2300.093812] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2301.951197] Key type lgssc registered [ 2303.838929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2319.914943] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2319.961718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2335.711564] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2340.835475] LustreError: 58474:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 2367.585213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.120385] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2387.438143] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2397.361320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.199378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.285846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2403.312864] LustreError: 59513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.327273] LustreError: 59513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2403.709251] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2408.420375] LustreError: 59511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.449480] LustreError: 59511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2413.542437] LustreError: 59514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.565837] LustreError: 59514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2414.494649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.943669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2420.520734] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2432.746991] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2442.270297] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2445.799450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.799450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.802020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2445.811413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2445.838195] Lustre: Skipped 1 previous similar message [ 2450.937877] LustreError: 60955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.960594] LustreError: 60955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2451.328602] Lustre: server umount lustre-OST0000 complete [ 2476.512253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2476.521677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2476.543542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.608483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.608954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.835842] Lustre: server umount lustre-MDT0000 complete [ 2485.740483] LustreError: 59517:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.761879] LustreError: 59517:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2487.487792] LustreError: 58480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788595 with bad export cookie 14553178455063631536 [ 2487.494277] LustreError: 58480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2487.494694] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.564681] Lustre: server umount lustre-MDT0001 complete [ 2510.203747] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2514.086951] Key type lgssc unregistered [ 2514.543621] LNet: 62225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.559491] LNetError: 62225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2514.589216] LNet: Removed LNI 192.168.203.159@tcp [ 2516.075251] Key type .llcrypt unregistered [ 2516.076779] Key type ._llcrypt unregistered [ 2549.007554] Key type ._llcrypt registered [ 2549.009810] Key type .llcrypt registered [ 2549.080252] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2550.505432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2550.547787] alg: No test for adler32 (adler32-zlib) [ 2551.523151] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2551.684511] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2553.311138] Key type lgssc registered [ 2554.920296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2566.815613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2566.866329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.564921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2574.198265] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2578.920528] LustreError: 63642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.956965] LustreError: 63642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2584.041529] LustreError: 63643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.236708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.000562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2593.811930] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2610.054743] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2621.919426] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2632.025876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2632.588313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.696665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2640.211611] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2649.349453] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2669.537673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2669.551188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.583626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2672.610613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2672.615148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2672.615864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2675.109468] Lustre: server umount lustre-OST0000 complete [ 2682.869919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.895081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.903312] Lustre: Skipped 1 previous similar message [ 2687.970411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2687.978366] Lustre: Skipped 1 previous similar message [ 2693.092748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2695.136342] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2695.572575] Lustre: server umount lustre-MDT0000 complete [ 2698.214311] LustreError: 63642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.269681] LustreError: 63642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2700.402993] LustreError: 63624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785788808 with bad export cookie 4346419002774004656 [ 2700.427080] LustreError: 63624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2700.428835] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2701.149078] Lustre: server umount lustre-MDT0001 complete [ 2717.079284] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2720.605527] Key type lgssc unregistered [ 2721.031947] LNet: 67212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2721.046397] LNetError: 67212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2721.079624] LNet: Removed LNI 192.168.203.159@tcp [ 2722.390397] Key type .llcrypt unregistered [ 2722.395290] Key type ._llcrypt unregistered [ 2755.560494] Key type ._llcrypt registered [ 2755.564299] Key type .llcrypt registered [ 2755.717758] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2757.456232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2757.691669] alg: No test for adler32 (adler32-zlib) [ 2759.108217] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2759.591946] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 2761.311117] Key type lgssc registered [ 2762.864855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2774.536887] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 16:27:59 (1785788879) [ 2782.412282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2782.450269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.957298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.335615] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2793.443798] LustreError: 68923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.472034] LustreError: 68923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2798.579129] LustreError: 68922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.819997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.630947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2808.392325] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2817.993976] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2826.933910] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2838.288895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2838.959817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2848.387213] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2854.437519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2857.350988] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2865.113846] Lustre: Failing over lustre-MDT0000 [ 2865.548635] Lustre: server umount lustre-MDT0000 complete [ 2869.217564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2869.231393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.281778] LustreError: 68922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.733795] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2874.540529] LustreError: 68916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.590502] LustreError: 68916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2877.998070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2878.309149] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2878.640327] Lustre: *** cfs_fail_loc=135, val=0*** [ 2878.840152] LustreError: 72012:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2878.854629] LustreError: 72012:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.159@tcp: cfg command failed: rc = -2 [ 2878.871477] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2878.871477] [ 2878.892284] LustreError: MGC192.168.203.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2878.917682] LustreError: 72000:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2878.943233] LustreError: 72000:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2878.966300] LustreError: 72000:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2878.979839] LustreError: 72000:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2879.029923] Lustre: server umount lustre-MDT0000 complete [ 2879.041539] LustreError: 72000:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2879.656633] LustreError: 68916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.688908] LustreError: 68916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2889.934896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.948437] LustreError: 68917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2889.979938] LustreError: 68917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2890.227036] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2890.254218] LustreError: 68915:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2890.750646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.848045] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2890.974762] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2895.882037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2895.978131] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2896.046639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2897.183515] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 2909.672847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.678524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.688910] Lustre: Skipped 1 previous similar message [ 2909.694290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2911.206240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2914.933579] Lustre: server umount lustre-OST0000 complete [ 2921.440498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.476547] Lustre: Skipped 1 previous similar message [ 2921.505258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.523747] Lustre: Skipped 2 previous similar messages [ 2926.561826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2926.578379] Lustre: Skipped 1 previous similar message [ 2927.241226] Lustre: server umount lustre-MDT0000 complete [ 2931.682540] LustreError: 68918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.711348] LustreError: 68918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2932.221460] LustreError: 70768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789039 with bad export cookie 4494334407900550749 [ 2932.230626] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2932.234258] LustreError: 70768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2933.211587] Lustre: server umount lustre-MDT0001 complete [ 2951.586660] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 2955.550902] Key type lgssc unregistered [ 2956.125683] LNet: 73817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2956.146749] LNetError: 73817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2956.166889] LNet: Removed LNI 192.168.203.159@tcp [ 2958.490754] Key type .llcrypt unregistered [ 2958.505054] Key type ._llcrypt unregistered [ 2994.650699] Key type ._llcrypt registered [ 2994.652553] Key type .llcrypt registered [ 2994.887825] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 2996.981105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2997.225802] alg: No test for adler32 (adler32-zlib) [ 2998.478645] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 2998.953330] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3000.786245] Key type lgssc registered [ 3002.969925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3016.579430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3016.592755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.227750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.287402] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3027.435736] LustreError: 75233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.454884] LustreError: 75233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3032.565233] LustreError: 75234:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3033.290829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.814493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3039.653116] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3049.185968] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3057.463819] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3065.742962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3066.059518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.985563] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3083.962089] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3088.708360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 3095.050687] Lustre: Failing over lustre-MDT0000 [ 3095.255213] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.59@tcp (stopping) [ 3095.873468] Lustre: server umount lustre-MDT0000 complete [ 3099.108850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.112200] LustreError: 75228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.116412] Lustre: Skipped 1 previous similar message [ 3099.133092] LustreError: 75228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3104.225962] LustreError: 75230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.255715] LustreError: 75230:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3107.414637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.684958] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3108.044979] Lustre: *** cfs_fail_loc=a05, val=0*** [ 3108.283260] LustreError: 78324:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 3108.314246] LustreError: 78324:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.159@tcp: cfg command failed: rc = -9 [ 3108.336133] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 3108.336133] [ 3108.358785] LustreError: MGC192.168.203.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3108.388149] LustreError: 78313:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 3108.401181] LustreError: 78313:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 3108.421219] LustreError: 78313:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3108.435605] LustreError: 78313:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 3108.493180] Lustre: server umount lustre-MDT0000 complete [ 3108.497949] LustreError: 78313:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 3109.346563] LustreError: 77544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.378833] LustreError: 77544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3118.412086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.574115] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3118.588093] LustreError: 75226:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3118.599196] LustreError: 78672:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.159@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000e4a61c9f) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3118.628675] LustreError: 75961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.649795] LustreError: 75961:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3118.947304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3119.000887] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3120.870825] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3124.202249] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3124.247715] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3124.311453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 3124.592091] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3139.577576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.590819] Lustre: Skipped 1 previous similar message [ 3139.599624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3139.610769] Lustre: Skipped 1 previous similar message [ 3141.737850] Lustre: server umount lustre-OST0000 complete [ 3149.793855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3149.804340] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.813088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.819562] Lustre: Skipped 2 previous similar messages [ 3152.761892] Lustre: server umount lustre-MDT0000 complete [ 3154.921929] LustreError: 75961:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3156.979851] LustreError: 75215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789264 with bad export cookie 1134135818702505080 [ 3156.980820] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3156.988212] LustreError: 75215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3157.482162] Lustre: server umount lustre-MDT0001 complete [ 3171.167824] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3174.987173] Key type lgssc unregistered [ 3175.354648] LNet: 80109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3175.364372] LNetError: 80109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3175.386288] LNet: Removed LNI 192.168.203.159@tcp [ 3176.527493] Key type .llcrypt unregistered [ 3176.530407] Key type ._llcrypt unregistered [ 3192.189327] Key type ._llcrypt registered [ 3192.193179] Key type .llcrypt registered [ 3192.278592] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3192.489401] Key type .llcrypt unregistered [ 3192.493282] Key type ._llcrypt unregistered [ 3223.605627] Key type ._llcrypt registered [ 3223.609279] Key type .llcrypt registered [ 3223.841172] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3224.958860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3224.990926] alg: No test for adler32 (adler32-zlib) [ 3226.491153] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3226.832272] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3228.599218] Key type lgssc registered [ 3230.002969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3239.321641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3239.356745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3241.072378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3246.767881] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3250.663766] LustreError: 82014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.680431] LustreError: 82014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3255.777249] LustreError: 82013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.630263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.038587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3263.562310] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3272.551297] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3279.134856] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3287.203699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3287.700930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.509510] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3303.142593] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3311.308448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 3326.963887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3326.967953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3327.003503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3327.850638] Lustre: server umount lustre-OST0000 complete [ 3334.625067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3334.648899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.690530] Lustre: Skipped 1 previous similar message [ 3334.705945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3334.721254] Lustre: Skipped 1 previous similar message [ 3337.184902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3337.187187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.805111] Lustre: server umount lustre-MDT0000 complete [ 3342.309425] LustreError: 83449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3342.323722] LustreError: 83449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3345.222460] LustreError: 81993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789452 with bad export cookie 7731738146649137217 [ 3345.246779] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3345.982636] Lustre: server umount lustre-MDT0001 complete [ 3359.796918] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3363.475423] Key type lgssc unregistered [ 3363.776279] LNet: 85569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3363.790088] LNetError: 85569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3363.817021] LNet: Removed LNI 192.168.203.159@tcp [ 3365.001176] Key type .llcrypt unregistered [ 3365.002987] Key type ._llcrypt unregistered [ 3397.487933] Key type ._llcrypt registered [ 3397.493250] Key type .llcrypt registered [ 3397.664982] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3400.021249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3400.052746] alg: No test for adler32 (adler32-zlib) [ 3401.294361] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3401.648837] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3403.431153] Key type lgssc registered [ 3405.316874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3418.639020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3418.715315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.232163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3429.355771] LustreError: 86987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.398927] LustreError: 86987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3430.024407] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3434.492917] LustreError: 86988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.608205] LustreError: 86987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.759706] LustreError: 86988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.828822] LustreError: 86987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3450.178091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3450.740802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3456.779874] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3467.814143] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3475.641573] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3484.862610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3485.446266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.518181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 3493.232189] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3502.966183] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3510.246344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3510.260647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.280808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3511.268050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3511.275419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3511.290853] Lustre: Skipped 1 previous similar message [ 3512.162886] Lustre: server umount lustre-OST0000 complete [ 3517.408682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3517.417906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.439406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3521.505490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3521.509265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3523.097632] Lustre: server umount lustre-MDT0000 complete [ 3526.630209] LustreError: 89433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.655812] LustreError: 89433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3527.469548] LustreError: 86969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789635 with bad export cookie 11331289638284801132 [ 3527.484551] LustreError: 86969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3527.490171] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3528.009603] Lustre: server umount lustre-MDT0001 complete [ 3542.967766] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3546.654869] Key type lgssc unregistered [ 3547.132709] LNet: 90524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3547.139319] LNetError: 90524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3547.157369] LNet: Removed LNI 192.168.203.159@tcp [ 3548.161594] Key type .llcrypt unregistered [ 3548.165100] Key type ._llcrypt unregistered [ 3582.156331] Key type ._llcrypt registered [ 3582.160291] Key type .llcrypt registered [ 3582.323715] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3584.155063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3584.437769] alg: No test for adler32 (adler32-zlib) [ 3585.685467] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3586.245537] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3588.071144] Key type lgssc registered [ 3589.233773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3598.076674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3598.108193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3599.806356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3605.183776] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3610.110425] LustreError: 91942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3610.155959] LustreError: 91942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3615.206754] LustreError: 91943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3615.951320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3616.579853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3622.033191] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3630.096389] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3636.465809] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3645.346767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3645.854522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3653.346150] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3662.716796] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3669.427942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3684.833820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3684.836548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3684.859440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3689.431307] Lustre: server umount lustre-OST0000 complete [ 3694.147316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3694.156704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3694.165921] Lustre: Skipped 2 previous similar messages [ 3694.190876] Lustre: Skipped 2 previous similar messages [ 3699.168420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3701.003745] Lustre: server umount lustre-MDT0000 complete [ 3704.288819] LustreError: 91943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3704.305457] LustreError: 91943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3705.263815] LustreError: 91923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785789812 with bad export cookie 13700779291509122908 [ 3705.281244] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3705.283487] LustreError: 91923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3706.093097] Lustre: server umount lustre-MDT0001 complete [ 3721.098395] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3724.407659] Key type lgssc unregistered [ 3724.794652] LNet: 95499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3724.809592] LNetError: 95499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3725.862322] LNet: Removed LNI 192.168.203.159@tcp [ 3726.852715] Key type .llcrypt unregistered [ 3726.854178] Key type ._llcrypt unregistered [ 3752.551306] Key type ._llcrypt registered [ 3752.552873] Key type .llcrypt registered [ 3752.722538] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 3754.237298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3754.495135] alg: No test for adler32 (adler32-zlib) [ 3755.651463] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 3755.975264] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 3757.697306] Key type lgssc registered [ 3758.670774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3767.447094] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3767.484712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3783.071510] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3788.191275] LustreError: 96887:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 3814.943432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3823.755332] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3833.126861] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3839.601483] Lustre: server umount lustre-OST0000 complete [ 3848.813220] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 16:45:54 (1785789954) [ 3856.614444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.294914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3862.502354] LustreError: 98529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3862.520863] LustreError: 98529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3863.089794] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3866.611771] LustreError: 98530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.713634] LustreError: 98529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3873.276081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.750603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3879.859431] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3888.809979] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3896.004797] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3904.190071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3904.575960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3911.465643] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 3919.625297] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3927.086015] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3944.418965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3944.431280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.454775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3947.492219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3947.503486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3947.524322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3947.551943] Lustre: Skipped 1 previous similar message [ 3949.074403] Lustre: server umount lustre-OST0000 complete [ 3955.681570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3955.690634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3955.709514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3957.730667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3957.732519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.687123] Lustre: server umount lustre-MDT0000 complete [ 3962.855622] LustreError: 98523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3962.887026] LustreError: 98523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3963.782221] LustreError: 98511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790071 with bad export cookie 12398004279602048757 [ 3963.799509] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3964.432337] Lustre: server umount lustre-MDT0001 complete [ 3977.082365] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 3980.536376] Key type lgssc unregistered [ 3980.947533] LNet: 102245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3980.957257] LNetError: 102245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3980.981810] LNet: Removed LNI 192.168.203.159@tcp [ 3982.117092] Key type .llcrypt unregistered [ 3982.118548] Key type ._llcrypt unregistered [ 4031.765470] Key type ._llcrypt registered [ 4031.766850] Key type .llcrypt registered [ 4031.838813] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4033.021495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4033.182828] alg: No test for adler32 (adler32-zlib) [ 4034.360737] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 4034.719711] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4036.472492] Key type lgssc registered [ 4038.146722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4046.936617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4046.969764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.530314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.303871] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4059.113517] LustreError: 104435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4059.131199] LustreError: 104435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4061.575773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.011173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4067.418523] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4076.019392] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4083.017136] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4090.635025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4091.008280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4094.065852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 4097.900578] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4106.308874] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4114.916241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4114.935197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.949957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4116.964831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4116.965098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.318604] Lustre: server umount lustre-OST0000 complete [ 4127.201299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4127.205498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4127.219777] Lustre: Skipped 1 previous similar message [ 4127.236363] Lustre: Skipped 2 previous similar messages [ 4130.448978] Lustre: server umount lustre-MDT0000 complete [ 4132.327768] LustreError: 104435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4132.347882] LustreError: 104435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4134.306876] LustreError: 104416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790242 with bad export cookie 12538770200511990936 [ 4134.310479] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4134.316897] LustreError: 104416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4134.804293] Lustre: server umount lustre-MDT0001 complete [ 4147.198269] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4150.459547] Key type lgssc unregistered [ 4150.860619] LNet: 107970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4150.866504] LNetError: 107970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4150.885861] LNet: Removed LNI 192.168.203.159@tcp [ 4151.942791] Key type .llcrypt unregistered [ 4151.947448] Key type ._llcrypt unregistered [ 4188.100263] Key type ._llcrypt registered [ 4188.101868] Key type .llcrypt registered [ 4188.180782] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4191.704893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4191.733100] alg: No test for adler32 (adler32-zlib) [ 4192.694278] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 4192.920763] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4194.599452] Key type lgssc registered [ 4197.450303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4213.765943] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4213.859591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4229.855649] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4234.975626] LustreError: 109215:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 4261.731565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4271.229570] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4283.261441] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4294.258211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.748312] LustreError: MGC192.168.203.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 4294.782871] LustreError: 110244:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 4294.811710] LustreError: 110244:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4294.818159] LustreError: 110244:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 4294.967632] Lustre: server umount lustre-MDT0000 complete [ 4294.970725] LustreError: 110244:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4313.055688] Lustre: 108552:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785790404/real 1785790404] req@00000000997c58dd x1872536849616256/t0(0) o400->MGC192.168.203.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785790420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4313.103926] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4313.125609] LustreError: 109229:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 4333.774090] Lustre: server umount lustre-OST0000 complete [ 4346.891347] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4362.422135] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4405.986576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4406.459763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4406.533662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4406.686099] Lustre: lustre-MDT0000: new disk, initializing [ 4406.807685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4406.840641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4412.652784] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4425.653555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.857096] Lustre: 113066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4425.946972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4425.953422] Lustre: Skipped 1 previous similar message [ 4426.067298] Lustre: lustre-MDT0001: new disk, initializing [ 4426.218182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4426.273489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4426.293166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4432.374538] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4445.562885] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4453.804988] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4462.440742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4462.735800] Lustre: lustre-OST0000: new disk, initializing [ 4462.739093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4462.746430] Lustre: 115033:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4462.833654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.602705] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4472.405630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4472.417212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4472.579213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4484.264717] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4490.916498] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4491.177755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4497.302183] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4497.744225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4503.527393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4503.531516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4503.531659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4503.563258] Lustre: Skipped 1 previous similar message [ 4503.595977] LustreError: Skipped 1 previous similar message [ 4505.961359] Lustre: server umount lustre-OST0000 complete [ 4513.253047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4513.268427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4513.296945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4513.310819] Lustre: Skipped 1 previous similar message [ 4516.790205] Lustre: server umount lustre-MDT0000 complete [ 4520.826057] LustreError: 113057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790628 with bad export cookie 1781290569856581133 [ 4520.837490] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4520.841031] LustreError: 113057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4521.610367] Lustre: server umount lustre-MDT0001 complete [ 4532.304588] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 16:57:17 (1785790637) [ 4536.958535] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 4557.134834] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4570.526081] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4617.241675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.535690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4617.583130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4617.683855] Lustre: lustre-MDT0000: new disk, initializing [ 4617.825215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4617.871363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4623.638423] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4636.271424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4636.458213] Lustre: 119938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4636.555463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4636.564636] Lustre: Skipped 1 previous similar message [ 4636.677924] Lustre: lustre-MDT0001: new disk, initializing [ 4636.823914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4636.894768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4636.906569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4641.910382] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4654.785461] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4661.502337] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4671.788351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4672.284498] Lustre: lustre-OST0000: new disk, initializing [ 4672.290542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4672.305419] Lustre: 121907:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4672.500097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4677.769819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4677.778984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4677.945769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4681.614926] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4693.377957] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4699.821309] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4700.154279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4705.683385] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4706.091842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4708.835816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4708.838304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4708.846753] LustreError: Skipped 1 previous similar message [ 4708.847781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4708.847784] Lustre: Skipped 1 previous similar message [ 4708.904321] Lustre: Skipped 2 previous similar messages [ 4713.955089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4713.965872] Lustre: Skipped 2 previous similar messages [ 4714.868643] Lustre: server umount lustre-OST0000 complete [ 4723.686792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4723.705367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.737197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4727.318291] Lustre: server umount lustre-MDT0000 complete [ 4729.313142] LustreError: 119946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.343754] LustreError: 119946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4732.180427] LustreError: 119932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790839 with bad export cookie 1781290569856582008 [ 4732.196034] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4732.202301] LustreError: 119932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4733.547203] Lustre: server umount lustre-MDT0001 complete [ 4746.083488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4747.095386] LustreError: 124129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.313332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.365347] LustreError: 124130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.718331] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4756.472058] LustreError: 124129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.591158] LustreError: 124130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4765.091727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4772.578461] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4782.569457] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4790.456693] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4799.986361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4807.150810] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 4815.613741] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4828.109321] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4836.321814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4836.334925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.365469] Lustre: Skipped 1 previous similar message [ 4836.380167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4836.382770] Lustre: Skipped 2 previous similar messages [ 4840.836602] Lustre: server umount lustre-OST0000 complete [ 4847.597816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4847.611935] LustreError: Skipped 1 previous similar message [ 4847.631150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.644472] Lustre: Skipped 1 previous similar message [ 4853.735302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4853.748215] Lustre: Skipped 6 previous similar messages [ 4860.385898] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4860.850879] Lustre: server umount lustre-MDT0000 complete [ 4863.977522] LustreError: 125060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.014181] LustreError: 125060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4866.448576] LustreError: 124109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785790974 with bad export cookie 1781290569856582897 [ 4866.450452] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4866.457505] LustreError: 124109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4867.258242] Lustre: server umount lustre-MDT0001 complete [ 4884.242346] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 4889.655620] Key type lgssc unregistered [ 4890.395457] LNet: 127774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4890.406751] LNetError: 127774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4891.429890] LNet: Removed LNI 192.168.203.159@tcp [ 4893.556835] Key type .llcrypt unregistered [ 4893.563965] Key type ._llcrypt unregistered [ 4927.898348] Key type ._llcrypt registered [ 4927.913405] Key type .llcrypt registered [ 4928.078961] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 4952.187432] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 4953.706910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4953.732882] alg: No test for adler32 (adler32-zlib) [ 4955.124362] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 4955.501243] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 4957.335209] Key type lgssc registered [ 4959.154791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5004.300963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5004.339739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.758428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5005.833402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5005.949782] Lustre: lustre-MDT0000: new disk, initializing [ 5006.000195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5006.025204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5012.632343] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5026.054510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.203208] Lustre: 130938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5026.239543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5026.243084] Lustre: Skipped 1 previous similar message [ 5026.296986] Lustre: lustre-MDT0001: new disk, initializing [ 5026.372489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5026.405261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5026.416979] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5032.735622] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5044.300109] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5053.603691] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5071.902368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5072.586733] Lustre: lustre-OST0000: new disk, initializing [ 5072.602157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5072.616266] Lustre: 132908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5072.983721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5080.275398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5080.324065] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5080.661621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5091.015361] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5111.502979] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5122.051193] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5122.465644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5131.824979] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5132.163846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5136.866612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5136.875164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.906995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5137.382515] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5137.434832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5137.458694] Lustre: Skipped 1 previous similar message [ 5142.384781] Lustre: server umount lustre-OST0000 complete [ 5152.737635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.744243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5152.756611] Lustre: Skipped 1 previous similar message [ 5152.787303] Lustre: Skipped 1 previous similar message [ 5157.381630] Lustre: server umount lustre-MDT0000 complete [ 5162.811260] LustreError: 130931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791270 with bad export cookie 2860387006676491682 [ 5162.819443] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5162.838661] LustreError: 130931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5164.110634] Lustre: server umount lustre-MDT0001 complete [ 5175.361431] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 17:08:00 (1785791280) [ 5185.661356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.358679] LustreError: 135326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5186.388499] LustreError: 135326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5186.535913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5191.657957] LustreError: 135327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5193.102534] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5196.771354] LustreError: 135326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.891899] LustreError: 135327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5203.646124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5204.230702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5209.917496] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5221.127199] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5229.935105] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5235.168069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5235.176299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5235.201072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5240.727611] Lustre: server umount lustre-MDT0000 complete [ 5242.336717] LustreError: 135322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5242.368481] LustreError: 135322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5245.801156] LustreError: 135308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791353 with bad export cookie 2860387006676492564 [ 5245.811426] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5245.820910] LustreError: 135308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5246.603995] Lustre: server umount lustre-MDT0001 complete [ 5259.688781] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 17:09:23 (1785791363) [ 5268.438319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5284.063728] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5289.183433] LustreError: 137746:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 5314.912375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5324.391831] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5334.551968] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5337.806503] Lustre: server umount lustre-OST0000 complete [ 5347.496440] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 17:10:53 (1785791453) [ 5356.249972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.676078] LustreError: 139195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.720310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5364.517926] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5376.481436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5383.087912] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5393.214881] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5402.346707] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5411.667978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5411.959775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.973143] Lustre: Skipped 1 previous similar message [ 5419.845961] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5429.236627] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5442.018050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.021243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5442.034672] Lustre: Skipped 3 previous similar messages [ 5442.051253] Lustre: Skipped 3 previous similar messages [ 5446.488036] Lustre: server umount lustre-MDT0000 complete [ 5447.139255] LustreError: 139925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5447.159890] LustreError: 139925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 5451.609442] LustreError: 139176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791559 with bad export cookie 2860387006676493229 [ 5451.612913] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5451.630331] LustreError: 139176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5452.325967] Lustre: server umount lustre-MDT0001 complete [ 5468.052578] Lustre: server umount lustre-OST0000 complete [ 5479.666907] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 17:13:03 (1785791583) [ 5488.311143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.029107] LustreError: 142851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5489.130178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5496.021826] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5507.641519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5514.047856] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5524.040356] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5532.005480] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5540.291503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5550.077252] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5559.291720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 5560.111558] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5567.641858] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5567.980448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5573.741505] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5573.927845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5578.720908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5578.729719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.752147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5578.755193] Lustre: Skipped 1 previous similar message [ 5583.519931] Lustre: server umount lustre-OST0000 complete [ 5590.011882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5595.104259] LustreError: 142852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.128130] LustreError: 142852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5595.257826] Lustre: server umount lustre-MDT0000 complete [ 5600.357152] LustreError: 142832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791708 with bad export cookie 2860387006676494300 [ 5600.370938] LustreError: 142832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5600.371618] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5600.975819] Lustre: server umount lustre-MDT0001 complete [ 5612.685432] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 17:15:17 (1785791717) [ 5622.757665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5638.434956] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5643.552250] LustreError: 147016:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 5669.280104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5669.283316] Lustre: Skipped 2 previous similar messages [ 5678.224921] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5687.842529] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5697.238785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.202927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 5704.320279] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5714.480115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5720.371888] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5729.866847] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5737.902978] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5745.955564] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5746.374766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5752.426440] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5752.860828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5756.393334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5756.402594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5756.445807] Lustre: Skipped 5 previous similar messages [ 5756.468839] Lustre: Skipped 6 previous similar messages [ 5763.731185] Lustre: server umount lustre-MDT0000 complete [ 5766.640857] LustreError: 149350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.685548] LustreError: 149350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 5770.659458] LustreError: 147022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785791878 with bad export cookie 2860387006676495322 [ 5770.693211] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5770.699351] LustreError: 147022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5771.733664] Lustre: server umount lustre-MDT0001 complete [ 5787.226712] Lustre: server umount lustre-OST0000 complete [ 5807.895159] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 17:18:31 (1785791911) [ 5823.900970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5839.776884] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5844.895416] LustreError: 151218:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 5879.893503] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5890.760166] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5898.823656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5899.247199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5903.636579] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5914.873752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.265507] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5928.887896] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5935.031287] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5943.685314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5943.931630] Lustre: lustre-OST0001: new disk, initializing [ 5943.939173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5943.950402] Lustre: 154095:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5944.129360] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5944.137884] Lustre: Skipped 5 previous similar messages [ 5949.975162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5949.986787] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5950.178366] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5952.053494] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 5972.118170] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5985.583386] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5986.525502] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5998.284879] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5998.556598] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6003.175285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6003.191139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6003.223615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6003.225646] Lustre: Skipped 3 previous similar messages [ 6016.995105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6017.275204] Lustre: server umount lustre-OST0000 complete [ 6027.250196] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6027.320664] LustreError: 151235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6027.357633] LustreError: 151235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6029.642528] Lustre: server umount lustre-OST0001 complete [ 6041.966937] Lustre: server umount lustre-MDT0000 complete [ 6047.808267] LustreError: 151226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792155 with bad export cookie 2860387006676496274 [ 6047.815289] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6047.820458] LustreError: 151226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6048.463678] Lustre: server umount lustre-MDT0001 complete [ 6082.429849] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 17:23:07 (1785792187) [ 6085.448280] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 6087.874390] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 6090.532896] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 17:23:15 (1785792195) [ 6100.187911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.747945] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6100.834046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6107.292270] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6119.071661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6119.212995] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6119.294708] Lustre: 157664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6128.128341] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6139.815906] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6147.658436] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6156.720918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6156.948809] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6162.626909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 6165.951422] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6178.730922] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6186.814667] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6187.252042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6195.205191] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6195.391778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6198.240639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6198.248996] LustreError: Skipped 1 previous similar message [ 6198.260275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.273971] Lustre: Skipped 5 previous similar messages [ 6198.285204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6198.300281] Lustre: Skipped 10 previous similar messages [ 6203.805616] Lustre: server umount lustre-OST0000 complete [ 6232.239383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6238.819491] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.841855] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6238.859694] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6240.107245] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6250.914463] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6257.309440] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6257.687490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6262.953865] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6263.168377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6270.498666] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9afe529e8000.ost_server_uuid 50 [ 6272.398118] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9afe529e8000.ost_server_uuid in FULL state after 0 sec [ 6272.692094] LustreError: 157671:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.59@tcp) returned error from blocking AST (req@000000003dcd3d9e x1872537649993984 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000ece91870/0x27b222d3dd1878ec lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.59@tcp remote: 0xcb15ccaf1fbadb92 expref: 5 pid: 158572 timeout: 6372 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6272.763485] LustreError: lustre-MDT0000: A client on nid 192.168.203.59@tcp was evicted due to a lock blocking callback time out: rc -107 [ 6272.784133] LustreError: 157661:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.59@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000ece91870/0x27b222d3dd1878ec lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.59@tcp remote: 0xcb15ccaf1fbadb92 expref: 6 pid: 158572 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6279.149929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6279.162349] LustreError: Skipped 1 previous similar message [ 6285.389785] Lustre: server umount lustre-OST0000 complete [ 6304.223273] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6304.665069] Lustre: server umount lustre-MDT0000 complete [ 6310.114662] LustreError: 159530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792417 with bad export cookie 2860387006676497457 [ 6310.128964] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6310.129684] LustreError: 159530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6317.056473] Lustre: server umount lustre-MDT0001 complete [ 6333.150293] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6337.198566] Key type lgssc unregistered [ 6337.608198] LNet: 163797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6337.618190] LNetError: 163797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6337.645577] LNet: Removed LNI 192.168.203.159@tcp [ 6338.953293] Key type .llcrypt unregistered [ 6338.957393] Key type ._llcrypt unregistered [ 6369.328273] Key type ._llcrypt registered [ 6369.331396] Key type .llcrypt registered [ 6369.493187] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6371.514789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6371.550040] alg: No test for adler32 (adler32-zlib) [ 6372.845789] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 6373.291642] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 6375.087162] Key type lgssc registered [ 6376.913123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6387.921873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6387.970640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.936316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6396.370513] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6399.472308] LustreError: 165212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.499960] LustreError: 165212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6404.577948] LustreError: 165213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6409.697949] LustreError: 165212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6410.063474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6410.518216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6417.899166] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6427.580832] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6436.268606] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6444.614584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6445.308849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.058991] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6459.657779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6467.145746] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6475.279938] Lustre: Failing over lustre-MDT0000 [ 6476.223249] Lustre: server umount lustre-MDT0000 complete [ 6477.291686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6477.303729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6477.327448] LustreError: 165213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6480.368165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6485.474233] LustreError: 165209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6485.512895] LustreError: 165209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6489.411078] LustreError: 165194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.59@tcp arrived at 1785792597 with bad export cookie 13572941916124870566 [ 6489.437944] LustreError: 165194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6495.718773] LustreError: 165212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6495.744156] LustreError: 165212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6496.735390] Lustre: 164521:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785792588/real 1785792588] req@00000000edda6df5 x1872539135540992/t0(0) o400->MGC192.168.203.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785792604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6496.775394] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6498.703874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6506.980648] LustreError: 164520:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006015311b x1872539135546368/t0(0) o250->MGC192.168.203.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6507.250167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6507.300394] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6512.611317] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6512.620631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6514.414238] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6518.066182] Lustre: lustre-MDT0000: Denying connection for new client 5aea723c-56ec-4b33-ba33-e58b9d564c42 (at 192.168.203.59@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 6527.473675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6527.488135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.513144] Lustre: Skipped 1 previous similar message [ 6527.528243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6533.099307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6533.106660] Lustre: Skipped 2 previous similar messages [ 6533.645395] Lustre: server umount lustre-OST0000 complete [ 6539.665331] LustreError: 168951:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 6539.673659] Lustre: 168299:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6539.688172] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 6539.716432] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 6539.733952] LustreError: 168299:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007ed99624 x1872539135562752/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6539.787408] LustreError: 168299:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 6539.803916] LustreError: 168299:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 6539.819819] Lustre: lustre-MDT0000: Recovery over after 0:27, of 2 clients 0 recovered and 2 were evicted. [ 6540.187360] Lustre: server umount lustre-MDT0000 complete [ 6543.330875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.366311] Lustre: Skipped 1 previous similar message [ 6543.383129] LustreError: 165208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.438707] LustreError: 165208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6547.231910] LustreError: 165194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792654 with bad export cookie 13572941916124870664 [ 6547.258728] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6554.112615] Lustre: server umount lustre-MDT0001 complete [ 6569.674801] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6573.591548] Key type lgssc unregistered [ 6574.084078] LNet: 169705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6574.110635] LNetError: 169705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6574.148270] LNet: Removed LNI 192.168.203.159@tcp [ 6575.710748] Key type .llcrypt unregistered [ 6575.713182] Key type ._llcrypt unregistered [ 6608.101480] Key type ._llcrypt registered [ 6608.108160] Key type .llcrypt registered [ 6608.288782] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6609.672638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6609.819879] alg: No test for adler32 (adler32-zlib) [ 6611.129636] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 6611.457510] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 6613.215150] Key type lgssc registered [ 6614.830109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6626.601379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6626.624305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.120511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.547914] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6637.537807] LustreError: 171135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6637.575145] LustreError: 171135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6642.661323] LustreError: 171136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6646.580395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.042109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6652.741630] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6663.286382] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6672.503223] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6682.110132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6682.764977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6691.800302] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6699.249913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6704.700251] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6714.848904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6714.858128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6714.880687] Lustre: Skipped 1 previous similar message [ 6714.906421] Lustre: Skipped 1 previous similar message [ 6719.979816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6719.984260] Lustre: Skipped 1 previous similar message [ 6720.697597] Lustre: server umount lustre-OST0000 complete [ 6729.184623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6729.205235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6729.253217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6730.216806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6735.226842] Lustre: server umount lustre-MDT0000 complete [ 6735.328204] LustreError: 172086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6735.371366] LustreError: 172086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6740.451202] LustreError: 171136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6740.496672] LustreError: 171136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6740.815328] LustreError: 171115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785792848 with bad export cookie 17433994261578044678 [ 6740.832794] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6740.845307] LustreError: 171115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6742.463885] Lustre: server umount lustre-MDT0001 complete [ 6764.976774] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6769.893235] Key type lgssc unregistered [ 6770.329011] LNet: 174697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6770.337721] LNetError: 174697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6770.378891] LNet: Removed LNI 192.168.203.159@tcp [ 6771.763177] Key type .llcrypt unregistered [ 6771.766039] Key type ._llcrypt unregistered [ 6816.087406] Key type ._llcrypt registered [ 6816.093975] Key type .llcrypt registered [ 6816.241912] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 6817.616838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6817.810246] alg: No test for adler32 (adler32-zlib) [ 6818.937218] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 6819.569305] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 6821.399154] Key type lgssc registered [ 6823.081184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6835.563343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6835.594779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6837.153181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6843.945522] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6846.532919] LustreError: 176402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6846.588156] LustreError: 176402:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6851.554139] LustreError: 176403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.679265] LustreError: 176402:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.008145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6858.794694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6867.194860] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6888.928098] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6900.184913] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6911.125151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6911.321631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.472555] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 6932.041782] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6932.845041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 6953.440310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6953.444913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6953.475331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6953.509861] Lustre: Skipped 1 previous similar message [ 6956.061430] Lustre: server umount lustre-OST0000 complete [ 6963.681772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6963.689205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6963.692021] Lustre: Skipped 1 previous similar message [ 6968.086916] Lustre: server umount lustre-MDT0000 complete [ 6968.817483] LustreError: 177214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.862084] LustreError: 177214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6973.203947] LustreError: 176385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793080 with bad export cookie 7445193106754761345 [ 6973.210697] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6973.225566] LustreError: 176385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6974.095960] Lustre: server umount lustre-MDT0001 complete [ 6989.007578] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 6992.132544] Key type lgssc unregistered [ 6992.492174] LNet: 179963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6992.506182] LNetError: 179963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6992.517745] LNet: Removed LNI 192.168.203.159@tcp [ 6993.475823] Key type .llcrypt unregistered [ 6993.479957] Key type ._llcrypt unregistered [ 7027.188297] Key type ._llcrypt registered [ 7027.191478] Key type .llcrypt registered [ 7027.369618] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7029.599990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7029.708069] alg: No test for adler32 (adler32-zlib) [ 7032.027984] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 7032.723295] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7034.543628] Key type lgssc registered [ 7037.470732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7051.491850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7051.553206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.235510] Lustre: *** cfs_fail_loc=135, val=0*** [ 7053.415491] LustreError: 181491:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 7053.431373] LustreError: 181491:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.159@tcp: cfg command failed: rc = -2 [ 7053.442310] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7053.442310] [ 7053.452318] LustreError: MGC192.168.203.159@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7053.466124] LustreError: 181451:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 7053.483716] LustreError: 181451:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7053.498472] LustreError: 181451:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7053.509391] LustreError: 181451:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7053.599847] Lustre: server umount lustre-MDT0000 complete [ 7053.604052] LustreError: 181451:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7062.792669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.489912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7069.204448] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7073.760802] LustreError: 181862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.783065] LustreError: 181862:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7078.881823] LustreError: 181863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7080.373172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7080.909313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7087.380829] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7098.119614] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7105.388906] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7110.626069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.654160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7110.663112] Lustre: Skipped 1 previous similar message [ 7115.745663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7115.750382] Lustre: Skipped 1 previous similar message [ 7120.868062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7120.892852] Lustre: Skipped 1 previous similar message [ 7122.399274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7122.917567] Lustre: server umount lustre-MDT0000 complete [ 7125.985824] LustreError: 181862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7126.020563] LustreError: 181862:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7128.160663] LustreError: 181844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793235 with bad export cookie 17455220622571171401 [ 7128.196666] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7128.858954] Lustre: server umount lustre-MDT0001 complete [ 7149.228805] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7153.955603] Key type lgssc unregistered [ 7154.430704] LNet: 184207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7154.477824] LNetError: 184207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7154.507649] LNet: Removed LNI 192.168.203.159@tcp [ 7155.864642] Key type .llcrypt unregistered [ 7155.868822] Key type ._llcrypt unregistered [ 7188.324878] Key type ._llcrypt registered [ 7188.329058] Key type .llcrypt registered [ 7188.514889] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7188.886419] Key type .llcrypt unregistered [ 7188.893092] Key type ._llcrypt unregistered [ 7214.418642] Key type ._llcrypt registered [ 7214.422097] Key type .llcrypt registered [ 7214.576092] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7216.286673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7216.552754] alg: No test for adler32 (adler32-zlib) [ 7218.120993] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 7218.488166] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7220.311159] Key type lgssc registered [ 7222.063719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7231.873084] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7231.896641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7247.583363] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7252.703418] LustreError: 186343:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.159@tcp: failed processing log, type 4: rc = -110 [ 7279.392143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7287.054703] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7299.397351] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7310.432184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7311.937292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7312.115494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 7318.755773] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7319.010744] LustreError: 187380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7319.037838] LustreError: 187380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7324.134937] LustreError: 187664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7324.176478] LustreError: 187664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7329.265388] LustreError: 187825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.293304] LustreError: 187825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7332.439852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7333.082367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7340.756680] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7351.058284] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7359.768443] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7365.356504] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 7378.400239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7378.412645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.445659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7379.434694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7379.437114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7384.552682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7384.560750] Lustre: Skipped 2 previous similar messages [ 7389.151145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7389.612825] Lustre: server umount lustre-OST0000 complete [ 7399.904456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7399.911489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7399.946118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7399.965697] Lustre: Skipped 1 previous similar message [ 7402.103801] Lustre: server umount lustre-MDT0000 complete [ 7405.024115] LustreError: 187382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7405.043978] LustreError: 187382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7406.728035] LustreError: 186350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793514 with bad export cookie 4339402208603301607 [ 7406.729428] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7406.740011] LustreError: 186350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7407.228832] Lustre: server umount lustre-MDT0001 complete [ 7422.583858] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7425.905996] Key type lgssc unregistered [ 7426.503518] LNet: 190218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7426.522083] LNetError: 190218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7426.554530] LNet: Removed LNI 192.168.203.159@tcp [ 7428.564826] Key type .llcrypt unregistered [ 7428.574599] Key type ._llcrypt unregistered [ 7462.917607] Key type ._llcrypt registered [ 7462.927717] Key type .llcrypt registered [ 7463.086119] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7464.254581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7464.300127] alg: No test for adler32 (adler32-zlib) [ 7465.698216] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 7466.062767] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7467.856075] Key type lgssc registered [ 7469.415291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7479.446626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7479.483659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7481.247263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7487.729542] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7490.529771] LustreError: 191631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7490.547799] LustreError: 191631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7495.660279] LustreError: 191632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7498.633807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.208923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7505.652931] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7515.699576] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7523.396212] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7532.909029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.586476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7542.201867] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7551.029988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 7553.876934] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7560.792081] Lustre: Failing over lustre-MDT0000 [ 7561.187875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.202872] LustreError: 191626:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7561.231177] LustreError: 191626:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7561.615939] Lustre: server umount lustre-MDT0000 complete [ 7566.309544] LustreError: 191632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.335905] LustreError: 191632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7571.430070] LustreError: 191631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7571.461046] LustreError: 191631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7576.547366] Lustre: 190942:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785793668/real 1785793668] req@00000000ec2351e6 x1872540281632384/t0(0) o400->MGC192.168.203.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785793684 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7576.609382] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7581.666269] LustreError: 191628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.705275] LustreError: 191628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7584.702678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7587.163508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7587.228610] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7587.883173] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7592.422871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7592.459161] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7592.542381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 7593.321706] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7599.281240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 7604.436072] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 7623.137983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7623.167674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7623.188803] Lustre: Skipped 3 previous similar messages [ 7628.256219] LustreError: 193455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7628.292964] LustreError: 193455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7628.590943] Lustre: server umount lustre-OST0000 complete [ 7638.496738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7638.497581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7638.504276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7638.504284] Lustre: Skipped 1 previous similar message [ 7638.517045] Lustre: Skipped 1 previous similar message [ 7642.904961] Lustre: server umount lustre-MDT0000 complete [ 7648.442240] LustreError: 195217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785793756 with bad export cookie 10811947925862016832 [ 7648.443988] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7648.465782] LustreError: 195217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7649.693529] Lustre: server umount lustre-MDT0001 complete [ 7672.298223] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7676.679409] Key type lgssc unregistered [ 7677.089485] LNet: 196480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7677.107139] LNetError: 196480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7677.129878] LNet: Removed LNI 192.168.203.159@tcp [ 7678.954266] Key type .llcrypt unregistered [ 7678.955887] Key type ._llcrypt unregistered [ 7717.239368] Key type ._llcrypt registered [ 7717.257116] Key type .llcrypt registered [ 7717.442428] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7718.819965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7719.104333] alg: No test for adler32 (adler32-zlib) [ 7721.335617] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 7722.105978] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7724.103399] Key type lgssc registered [ 7727.151392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7744.869882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7744.936627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.478128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7756.875946] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7757.793920] LustreError: 197893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7757.868926] LustreError: 197893:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7762.943137] LustreError: 197894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.043993] LustreError: 197893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.806067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7773.160830] LustreError: 197894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7773.817527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7781.830712] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7793.864928] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7803.480512] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7815.394455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7816.190098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7818.339370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 7825.460605] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7835.103822] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7840.116860] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7846.424131] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 7870.729061] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7880.160596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7880.176422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.201231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7880.688014] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7880.709096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.760743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7885.091365] Lustre: server umount lustre-OST0000 complete [ 7890.912674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.919319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7890.938669] Lustre: Skipped 1 previous similar message [ 7896.034255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7896.045736] Lustre: Skipped 2 previous similar messages [ 7897.680777] Lustre: server umount lustre-MDT0000 complete [ 7901.162861] LustreError: 197890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7901.188316] LustreError: 197890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7902.770503] LustreError: 197876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794010 with bad export cookie 6432061163257013005 [ 7902.782254] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7902.788782] LustreError: 197876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7903.817514] Lustre: server umount lustre-MDT0001 complete [ 7918.740843] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 7923.253835] Key type lgssc unregistered [ 7923.630293] LNet: 201778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7923.653314] LNetError: 201778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7923.694730] LNet: Removed LNI 192.168.203.159@tcp [ 7925.261265] Key type .llcrypt unregistered [ 7925.272661] Key type ._llcrypt unregistered [ 7953.868213] Key type ._llcrypt registered [ 7953.872685] Key type .llcrypt registered [ 7954.055260] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 7956.308202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7956.364674] alg: No test for adler32 (adler32-zlib) [ 7957.522813] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 7957.973666] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 7959.743175] Key type lgssc registered [ 7961.270470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7971.153356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7971.169924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7972.920967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7979.082788] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 7983.075821] LustreError: 203047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7983.099759] LustreError: 203047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7988.209316] LustreError: 203048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7991.499512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7991.921928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7999.177479] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8011.353815] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8019.271076] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8029.447636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8030.134959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8040.331810] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8043.456122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 8050.500039] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8058.183064] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 8069.638239] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 8081.753221] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 8088.938435] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 8103.906629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8103.915075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8103.926417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8104.929678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8104.934293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8104.940238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8107.419831] Lustre: server umount lustre-OST0000 complete [ 8114.658595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8114.669842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.698357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8114.705591] Lustre: Skipped 1 previous similar message [ 8119.289446] Lustre: server umount lustre-MDT0000 complete [ 8120.290666] LustreError: 203787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8120.312895] LustreError: 203787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8124.529498] LustreError: 203029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794232 with bad export cookie 6465368852921106333 [ 8124.542150] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8124.548413] LustreError: 203029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8125.630322] Lustre: server umount lustre-MDT0001 complete [ 8140.909795] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 8145.720337] Key type lgssc unregistered [ 8146.130072] LNet: 207928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8146.147784] LNetError: 207928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8146.174839] LNet: Removed LNI 192.168.203.159@tcp [ 8147.433071] Key type .llcrypt unregistered [ 8147.434840] Key type ._llcrypt unregistered [ 8179.328292] Key type ._llcrypt registered [ 8179.330868] Key type .llcrypt registered [ 8179.416974] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 8180.702318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8180.721511] alg: No test for adler32 (adler32-zlib) [ 8181.850283] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 8182.118401] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 8183.847125] Key type lgssc registered [ 8185.449403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8195.090470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8195.132172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8196.837765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8204.452073] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8206.318027] LustreError: 209343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8206.352469] LustreError: 209343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8211.437268] LustreError: 209344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8216.304266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.558280] LustreError: 209343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8216.974250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8223.502786] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8234.703317] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8243.835619] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.786222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8253.070591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8261.074314] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8266.437072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 8272.903867] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8293.345753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8293.374807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8293.404652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8297.460422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8297.468344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8299.300894] Lustre: server umount lustre-OST0000 complete [ 8307.684182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8307.733139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8307.748980] Lustre: Skipped 2 previous similar messages [ 8312.171229] Lustre: server umount lustre-MDT0000 complete [ 8312.805216] LustreError: 209339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8312.831824] LustreError: 209339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8317.408513] LustreError: 209324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794425 with bad export cookie 14618683841570476107 [ 8317.413419] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8317.424337] LustreError: 209324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8317.758058] Lustre: server umount lustre-MDT0001 complete [ 8332.606304] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 8336.215027] Key type lgssc unregistered [ 8336.540663] LNet: 213036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8336.555470] LNetError: 213036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8336.590523] LNet: Removed LNI 192.168.203.159@tcp [ 8337.747330] Key type .llcrypt unregistered [ 8337.749065] Key type ._llcrypt unregistered [ 8368.075431] Key type ._llcrypt registered [ 8368.085837] Key type .llcrypt registered [ 8368.314038] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 8370.173577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8370.219500] alg: No test for adler32 (adler32-zlib) [ 8371.473527] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 8371.894811] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 8373.713071] Key type lgssc registered [ 8375.347167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8384.581311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8384.623375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8386.300789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.466399] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8395.752630] LustreError: 214316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8395.775173] LustreError: 214316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8400.880306] LustreError: 214317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8405.557064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8405.990657] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8406.228472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8413.248862] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8423.766432] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8432.711869] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8443.266835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8444.066781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8453.914356] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8456.314760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 8465.483424] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8487.399744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8487.413776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8487.415682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8487.437183] Lustre: Skipped 1 previous similar message [ 8487.462735] Lustre: Skipped 1 previous similar message [ 8491.758455] Lustre: server umount lustre-OST0000 complete [ 8497.632349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8497.669415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8497.682188] Lustre: Skipped 1 previous similar message [ 8502.753302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8502.768742] Lustre: Skipped 1 previous similar message [ 8504.406276] Lustre: server umount lustre-MDT0000 complete [ 8507.873207] LustreError: 214316:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8507.889991] LustreError: 214316:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8509.843218] LustreError: 214295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794617 with bad export cookie 5516917407274183007 [ 8509.855622] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8509.862457] LustreError: 214295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8510.960789] Lustre: server umount lustre-MDT0001 complete [ 8526.474461] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 8530.165042] Key type lgssc unregistered [ 8530.611583] LNet: 218006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8530.628391] LNetError: 218006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8530.675086] LNet: Removed LNI 192.168.203.159@tcp [ 8531.955300] Key type .llcrypt unregistered [ 8531.959490] Key type ._llcrypt unregistered [ 8558.465146] Key type ._llcrypt registered [ 8558.472418] Key type .llcrypt registered [ 8558.638745] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 8560.530031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8560.559309] alg: No test for adler32 (adler32-zlib) [ 8561.794102] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 8562.299611] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 8564.071681] Key type lgssc registered [ 8566.059438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8576.962951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8576.988301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8578.709153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8585.730945] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8589.281563] LustreError: 219277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.315570] LustreError: 219277:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8594.410430] LustreError: 219276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8597.124118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8597.700713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8604.116235] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8613.223705] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8621.068712] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8630.589469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8631.264849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8639.398542] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8648.701632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 8650.482601] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8663.579514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8663.843029] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8664.194042] Lustre: lustre-OST0001: Not available for connect from 192.168.203.59@tcp (not set up) [ 8664.472654] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8675.328821] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8686.155146] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8688.638486] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8700.984729] Lustre: Permanently deactivating lustre-OST0001 [ 8701.000567] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 8710.291387] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 8719.970601] Lustre: Permanently reactivating lustre-OST0001 [ 8720.015627] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 8720.018367] Lustre: Skipped 2 previous similar messages [ 8720.186196] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8720.232160] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 8720.249130] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8720.276880] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8720.376458] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8735.714587] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8735.726721] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8735.727235] LustreError: Skipped 1 previous similar message [ 8735.768176] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8735.775786] Lustre: Skipped 2 previous similar messages [ 8738.794282] Lustre: server umount lustre-OST0001 complete [ 8741.352324] LustreError: 221101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8741.386159] LustreError: 221101:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8744.928293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8744.945532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8744.982469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8744.984957] Lustre: Skipped 3 previous similar messages [ 8745.951917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8746.468222] LustreError: 221724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8746.506179] LustreError: 221724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8750.200604] Lustre: server umount lustre-OST0000 complete [ 8756.192190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8756.203891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8756.223795] Lustre: Skipped 1 previous similar message [ 8756.244200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8756.257361] Lustre: Skipped 3 previous similar messages [ 8761.848385] LustreError: 220005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.867764] LustreError: 220005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8761.939764] Lustre: server umount lustre-MDT0000 complete [ 8767.590259] LustreError: 219256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785794875 with bad export cookie 1330877557454780395 [ 8767.619182] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8768.659650] Lustre: server umount lustre-MDT0001 complete [ 8784.181299] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 8788.129448] Key type lgssc unregistered [ 8788.504215] LNet: 224330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8788.520871] LNetError: 224330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8788.573415] LNet: Removed LNI 192.168.203.159@tcp [ 8789.669843] Key type .llcrypt unregistered [ 8789.680107] Key type ._llcrypt unregistered [ 8846.609869] Key type ._llcrypt registered [ 8846.614787] Key type .llcrypt registered [ 8846.760369] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 8847.902760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8848.059677] alg: No test for adler32 (adler32-zlib) [ 8849.410975] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 8849.840859] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 8851.663339] Key type lgssc registered [ 8853.209772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8862.813460] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8862.845118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.253079] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8864.301776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8864.589500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8871.076206] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8882.504283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8882.640508] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8882.703601] Lustre: 226539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8883.140710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8889.352444] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8900.136367] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8910.849879] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8919.351283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8919.652518] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8919.953504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8928.545071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 8929.095767] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 8941.947831] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8946.566257] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 8950.948754] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 8960.263593] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 8968.462453] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 8977.276218] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 8995.715508] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 8995.728716] Lustre: Skipped 2 previous similar messages [ 9033.056295] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 9033.069324] Lustre: Skipped 4 previous similar messages [ 9098.022812] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 9098.032250] Lustre: Skipped 8 previous similar messages [ 9107.938352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9107.941449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.956070] LustreError: Skipped 1 previous similar message [ 9107.983113] Lustre: Skipped 1 previous similar message [ 9107.989870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9108.000706] Lustre: Skipped 1 previous similar message [ 9108.569272] Lustre: server umount lustre-OST0000 complete [ 9118.177831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9118.203906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9118.213811] Lustre: Skipped 1 previous similar message [ 9120.748743] Lustre: server umount lustre-MDT0000 complete [ 9125.772193] LustreError: 226530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785795233 with bad export cookie 6288116369363488582 [ 9125.775407] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9125.791141] LustreError: 226530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9126.590888] Lustre: server umount lustre-MDT0001 complete [ 9145.336427] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 9149.266262] Key type lgssc unregistered [ 9149.685715] LNet: 231353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9149.691606] LNetError: 231353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9149.724673] LNet: Removed LNI 192.168.203.159@tcp [ 9151.362877] Key type .llcrypt unregistered [ 9151.368234] Key type ._llcrypt unregistered [ 9183.702545] Key type ._llcrypt registered [ 9183.715446] Key type .llcrypt registered [ 9183.851453] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 9185.317641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9185.378248] alg: No test for adler32 (adler32-zlib) [ 9186.717342] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 9187.019729] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 9188.679585] Key type lgssc registered [ 9190.508327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9201.300957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9201.340968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9203.459426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9212.378649] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9213.945083] LustreError: 232770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9213.985713] LustreError: 232770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9219.041704] LustreError: 232769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9224.170242] LustreError: 232770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9224.813214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9225.466608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9233.026677] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9244.446161] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9255.340929] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9266.688092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9267.481531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9273.673933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 9276.629631] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_default_debug -1 all [ 9286.005186] Lustre: DEBUG MARKER: oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9311.713385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9311.733752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9311.771557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9314.279117] Lustre: server umount lustre-OST0000 complete [ 9322.978736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9322.999352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9323.020912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9325.024287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9325.025718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9330.146599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9330.158027] Lustre: Skipped 2 previous similar messages [ 9332.385785] Lustre: server umount lustre-MDT0000 complete [ 9336.289444] LustreError: 232764:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9336.316799] LustreError: 232764:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9338.094757] LustreError: 232749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785795445 with bad export cookie 14897527070185279763 [ 9338.097080] LustreError: MGC192.168.203.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9338.117124] LustreError: 232749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9338.723572] Lustre: server umount lustre-MDT0001 complete [ 9354.653654] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 9358.482133] Key type lgssc unregistered [ 9359.053048] LNet: 236482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9359.066172] LNetError: 236482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9359.099292] LNet: Removed LNI 192.168.203.159@tcp [ 9360.872974] Key type .llcrypt unregistered [ 9360.878923] Key type ._llcrypt unregistered [ 9394.652600] Key type ._llcrypt registered [ 9394.657941] Key type .llcrypt registered [ 9394.902980] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 9396.761215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9396.938670] alg: No test for adler32 (adler32-zlib) [ 9398.256340] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 9398.746352] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 9400.567690] Key type lgssc registered [ 9402.301742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9675.788988] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 9679.360593] Key type lgssc unregistered [ 9679.712550] LNet: 238506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9679.716921] LNetError: 238506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9679.746732] LNet: Removed LNI 192.168.203.159@tcp [ 9680.725507] Key type .llcrypt unregistered [ 9680.728590] Key type ._llcrypt unregistered [ 9716.635075] Key type ._llcrypt registered [ 9716.645519] Key type .llcrypt registered [ 9716.876905] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing set_hostid [ 9745.532378] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing load_modules_local [ 9747.066558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9747.652512] alg: No test for adler32 (adler32-zlib) [ 9749.062953] Lustre: Lustre: Build Version: 2.17.56_1_g3060c05 [ 9749.699852] LNet: Added LNI 192.168.203.159@tcp [8/256/0/180] [ 9751.607843] Key type lgssc registered [ 9754.335589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9796.684340] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9399 sec ========== 18:25:01 (1785795901) [ 9799.690138] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:25:03 (1785795903) === [ 9806.523635] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:25:10 (1785795910) === [ 9860.300473] Lustre: DEBUG MARKER: oleg359-server.virtnet: executing unload_modules_local [ 9865.061687] Key type lgssc unregistered [ 9865.396687] LNet: 243751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9865.412564] LNetError: 243751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9865.440729] LNet: Removed LNI 192.168.203.159@tcp [ 9867.154728] Key type .llcrypt unregistered [ 9867.160634] Key type ._llcrypt unregistered