[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475015906 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002231] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007317] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011060] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.014028] *** VALIDATE selinux *** [ 0.022694] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027087] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.031489] *** VALIDATE proc *** [ 0.032261] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034293] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036064] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042811] debug: unmapping init [mem 0xffffffffb4859000-0xffffffffb4860fff] [ 0.044197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051014] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053350] rcu: Hierarchical SRCU implementation. [ 0.055477] smp: Bringing up secondary CPUs ... [ 0.056614] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060510] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135576] node 0 deferred pages initialised in 71ms [ 0.138389] devtmpfs: initialized [ 0.139294] x86/mm: Memory block size: 128MB [ 0.140000] gcov: version magic: 0x41383552 [ 0.143139] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146256] pinctrl core: initialized pinctrl subsystem [ 0.147206] [ 0.147736] ************************************************************* [ 0.148013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149019] ** ** [ 0.150016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151013] ** ** [ 0.152010] ** This means that this kernel is built to expose internal ** [ 0.153011] ** IOMMU data structures, which may compromise security on ** [ 0.154014] ** your system. ** [ 0.155012] ** ** [ 0.156014] ** If you see this message and you are not debugging the ** [ 0.157015] ** kernel, report this immediately to your vendor! ** [ 0.158013] ** ** [ 0.159017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160024] ************************************************************* [ 0.161645] NET: Registered protocol family 16 [ 0.162476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165458] cpuidle: using governor menu [ 0.167194] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.168496] PCI: Using configuration type 1 for base access [ 0.169145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.176108] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.181138] cryptd: max_cpu_qlen set to 1000 [ 0.183089] ACPI: Added _OSI(Module Device) [ 0.184018] ACPI: Added _OSI(Processor Device) [ 0.185017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.186017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.190049] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.193683] ACPI: Interpreter enabled [ 0.194078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195018] ACPI: Using IOAPIC for interrupt routing [ 0.196116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.197462] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.206596] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.207034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.208015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.209109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.211383] acpiphp: Slot [2] registered [ 0.212139] acpiphp: Slot [5] registered [ 0.213117] acpiphp: Slot [6] registered [ 0.214118] acpiphp: Slot [7] registered [ 0.215126] acpiphp: Slot [8] registered [ 0.216113] acpiphp: Slot [9] registered [ 0.217134] acpiphp: Slot [10] registered [ 0.218115] acpiphp: Slot [3] registered [ 0.219098] acpiphp: Slot [4] registered [ 0.220112] acpiphp: Slot [11] registered [ 0.221105] acpiphp: Slot [12] registered [ 0.222099] acpiphp: Slot [13] registered [ 0.223134] acpiphp: Slot [14] registered [ 0.224199] acpiphp: Slot [15] registered [ 0.225111] acpiphp: Slot [16] registered [ 0.226099] acpiphp: Slot [17] registered [ 0.227105] acpiphp: Slot [18] registered [ 0.228102] acpiphp: Slot [19] registered [ 0.229126] acpiphp: Slot [20] registered [ 0.230140] acpiphp: Slot [21] registered [ 0.231113] acpiphp: Slot [22] registered [ 0.232086] acpiphp: Slot [23] registered [ 0.233090] acpiphp: Slot [24] registered [ 0.234065] acpiphp: Slot [25] registered [ 0.235045] acpiphp: Slot [26] registered [ 0.236097] acpiphp: Slot [27] registered [ 0.237187] acpiphp: Slot [28] registered [ 0.238143] acpiphp: Slot [29] registered [ 0.239138] acpiphp: Slot [30] registered [ 0.240132] acpiphp: Slot [31] registered [ 0.241072] PCI host bridge to bus 0000:00 [ 0.242021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.243048] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.244023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.245033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.246025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.247033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.248176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.250027] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.252299] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.258018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.261093] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.262027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.263020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.264018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.265575] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.266755] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.267042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.268745] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.271015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.279974] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.281989] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.285517] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.288016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.291016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.298020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.304511] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.307017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.310016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.317017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.322993] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.326016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.329016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.336017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.341822] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.345016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.348015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.356017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.362175] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.365021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.368013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.375013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.380279] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.383016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.386016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.394019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.400739] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.401386] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.402391] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.403344] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.404238] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.407152] iommu: Default domain type: Passthrough [ 0.408407] SCSI subsystem initialized [ 0.409097] ACPI: bus type USB registered [ 0.410026] usbcore: registered new interface driver usbfs [ 0.411055] usbcore: registered new interface driver hub [ 0.412090] usbcore: registered new device driver usb [ 0.413126] pps_core: LinuxPPS API ver. 1 registered [ 0.414008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.415059] PTP clock support registered [ 0.416145] EDAC MC: Ver: 3.0.0 [ 0.417327] PCI: Using ACPI for IRQ routing [ 0.418721] NetLabel: Initializing [ 0.419011] NetLabel: domain hash size = 128 [ 0.420013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.421128] NetLabel: unlabeled traffic allowed by default [ 0.422129] vgaarb: loaded [ 0.423265] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.424015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.428000] clocksource: Switched to clocksource kvm-clock [ 0.537456] VFS: Disk quotas dquot_6.6.0 [ 0.539052] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.541342] *** VALIDATE ramfs *** [ 0.542606] *** VALIDATE hugetlbfs *** [ 0.543628] pnp: PnP ACPI init [ 0.545704] pnp: PnP ACPI: found 6 devices [ 0.566795] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.570715] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.572311] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.573608] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.575896] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.578611] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.581319] NET: Registered protocol family 2 [ 0.583236] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.588331] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.594044] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.599463] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.603323] TCP: Hash tables configured (established 65536 bind 65536) [ 0.606525] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.609362] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.612251] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.615238] NET: Registered protocol family 1 [ 0.617494] RPC: Registered named UNIX socket transport module. [ 0.619736] RPC: Registered udp transport module. [ 0.621160] RPC: Registered tcp transport module. [ 0.622755] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.625147] NET: Registered protocol family 44 [ 0.626483] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.628504] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.630538] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.632606] PCI: CLS 0 bytes, default 64 [ 0.633926] Unpacking initramfs... [ 2.016631] debug: unmapping init [mem 0xffff97753cc54000-0xffff97753ffbffff] [ 2.020620] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.023039] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.025954] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.506425] Initialise system trusted keyrings [ 2.508240] Key type blacklist registered [ 2.510188] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.519767] zbud: loaded [ 2.522868] *** VALIDATE nfs *** [ 2.523989] *** VALIDATE nfs4 *** [ 2.525652] pstore: using deflate compression [ 2.529082] Platform Keyring initialized [ 2.623506] NET: Registered protocol family 38 [ 2.625497] Key type asymmetric registered [ 2.627183] Asymmetric key parser 'x509' registered [ 2.629109] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.632084] io scheduler mq-deadline registered [ 2.633629] io scheduler kyber registered [ 2.635089] io scheduler bfq registered [ 2.636771] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.639769] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.642626] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.645533] ACPI: Power Button [PWRF] [ 2.650406] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.657208] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.668175] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.673907] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.689801] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.717890] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.746984] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.750320] Non-volatile memory driver v1.3 [ 2.751238] Linux agpgart interface v0.103 [ 2.772521] virtio_blk virtio1: [vda] 147040 512-byte logical blocks (75.3 MB/71.8 MiB) [ 2.775448] vda: detected capacity change from 0 to 75284480 [ 2.794768] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.798162] vdb: detected capacity change from 0 to 1073741824 [ 2.813321] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.816038] vdc: detected capacity change from 0 to 2621440000 [ 2.839374] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.841947] vdd: detected capacity change from 0 to 2621440000 [ 2.855668] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.857408] vde: detected capacity change from 0 to 4294967296 [ 2.870417] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.873312] vdf: detected capacity change from 0 to 4294967296 [ 2.879604] libphy: Fixed MDIO Bus: probed [ 2.895910] usbcore: registered new interface driver usbserial_generic [ 2.898151] usbserial: USB Serial support registered for generic [ 2.899711] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.902783] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.903800] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.905272] mousedev: PS/2 mouse device common for all mice [ 2.907330] rtc_cmos 00:05: RTC can wake from S4 [ 2.909202] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.909954] rtc_cmos 00:05: registered as rtc0 [ 2.914048] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.914200] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.916991] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.918543] intel_pstate: CPU model not supported [ 2.923824] hid: raw HID events driver (C) Jiri Kosina [ 2.925592] usbcore: registered new interface driver usbhid [ 2.927226] usbhid: USB HID core driver [ 2.928542] drop_monitor: Initializing network drop monitor service [ 2.930502] Initializing XFRM netlink socket [ 2.932170] NET: Registered protocol family 10 [ 2.934610] Segment Routing with IPv6 [ 2.936187] NET: Registered protocol family 17 [ 2.937467] mpls_gso: MPLS GSO support [ 2.941956] RAS: Correctable Errors collector initialized. [ 2.943596] AVX version of gcm_enc/dec engaged. [ 2.944690] AES CTR mode by8 optimization enabled [ 3.012860] sched_clock: Marking stable (3012837136, 0)->(4082640405, -1069803269) [ 3.015714] registered taskstats version 1 [ 3.017361] Loading compiled-in X.509 certificates [ 3.019286] zswap: loaded using pool lzo/zbud [ 3.037035] Key type big_key registered [ 3.048438] Key type encrypted registered [ 3.049495] ima: No TPM chip found, activating TPM-bypass! [ 3.050681] ima: Allocated hash algorithm: sha1 [ 3.051583] ima: No architecture policies found [ 3.052537] evm: Initialising EVM extended attributes: [ 3.053631] evm: security.selinux [ 3.054408] evm: security.ima [ 3.054917] evm: security.capability [ 3.055862] evm: HMAC attrs: 0x1 [ 3.057356] rtc_cmos 00:05: setting system clock to 2026-08-07 03:48:33 UTC (1786074513) [ 3.061331] debug: unmapping init [mem 0xffffffffb5803000-0xffffffffb59fffff] [ 3.063138] debug: unmapping init [mem 0xffffffffb4582000-0xffffffffb4858fff] [ 3.070060] Write protecting the kernel read-only data: 28672k [ 3.072118] debug: unmapping init [mem 0xffffffffb2c03000-0xffffffffb2dfffff] [ 3.074024] debug: unmapping init [mem 0xffffffffb3514000-0xffffffffb35fffff] [ 3.099567] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.104931] systemd[1]: Detected virtualization kvm. [ 3.106251] systemd[1]: Detected architecture x86-64. [ 3.107351] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.130580] systemd[1]: No hostname configured. [ 3.132081] systemd[1]: Set hostname to . [ 3.133632] random: systemd: uninitialized urandom read (16 bytes read) [ 3.135346] systemd[1]: Initializing machine ID from random generator. [ 3.174736] random: ln: uninitialized urandom read (6 bytes read) [ 3.243384] random: systemd: uninitialized urandom read (16 bytes read) [ 3.246637] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.251422] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.255770] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.738757] device-mapper: uevent: version 1.0.3 [ 3.740836] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 4.281317] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.453666] virtio_net virtio0 ens2: renamed from eth0 [ 4.514018] scsi host0: ata_piix [ 4.552276] scsi host1: ata_piix [ 4.553842] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.556353] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.003245] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.561899] random: crng init done [ 9.563404] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.953772] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.110725] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.381893] SELinux: Disabled at runtime. [ 11.443149] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.451356] systemd[1]: Detected virtualization kvm. [ 11.453169] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.915667] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.918668] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.921894] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.924445] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.928241] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.940941] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.947839] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ 12.095711] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.392928] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.661801] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.786662] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.926581] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.935538] EDAC sbridge: Ver: 1.1.2 [ 14.411944] Key type dns_resolver registered [ 14.718130] NFS: Registering the id_resolver key type [ 14.720055] Key type id_resolver registered [ 14.721321] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg227-server login: [ 51.041134] hrtimer: interrupt took 2959033 ns [ 68.839549] libcfs: loading out-of-tree module taints kernel. [ 68.878736] Key type ._llcrypt registered [ 68.881058] Key type .llcrypt registered [ 69.041468] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 93.151352] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 95.503655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.536710] alg: No test for adler32 (adler32-zlib) [ 97.393520] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 98.398378] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 100.199977] Key type lgssc registered [ 102.166666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 123.760710] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 169.458553] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 182.717950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 182.773681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.339352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.437205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.607345] Lustre: lustre-MDT0000: new disk, initializing [ 184.727092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.761771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.136462] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.268532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.380678] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 205.407883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 205.415518] Lustre: Skipped 1 previous similar message [ 205.539575] Lustre: lustre-MDT0001: new disk, initializing [ 205.619954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.674703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.684717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.831658] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.273587] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 225.564223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.850283] Lustre: lustre-OST0000: new disk, initializing [ 225.853606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 225.859895] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.929338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.318784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.326897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 229.428447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.336043] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.384606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.513253] Lustre: lustre-OST0001: new disk, initializing [ 248.524784] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 248.532974] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.592883] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.010303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 255.019710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 255.148068] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.867426] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.027252] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.946278] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 286.686335] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing check_logdir /tmp/testlogs/ [ 293.000982] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing yml_node [ 297.924125] Lustre: DEBUG MARKER: Client: 2.17.56.3 [ 301.188503] Lustre: DEBUG MARKER: MDS: 2.17.56.3 [ 303.739926] Lustre: DEBUG MARKER: OSS: 2.17.56.3 [ 305.590122] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 6 23:53:34 EDT 2026 [ 325.214269] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 326.896208] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 344.032561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 344.040560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.068445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.104189] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.108121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.165517] Lustre: Skipped 3 previous similar messages [ 349.602607] Lustre: server umount lustre-MDT0000 complete [ 354.218446] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786074864 with bad export cookie 11670483874441563309 [ 354.225314] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 354.225782] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 357.344208] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.344453] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.378046] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 357.392552] Lustre: Skipped 2 previous similar messages [ 357.396231] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 357.407676] Lustre: Skipped 1 previous similar message [ 359.396200] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.402256] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.437944] Lustre: Skipped 1 previous similar message [ 360.690503] Lustre: server umount lustre-MDT0001 complete [ 364.831808] Lustre: server umount lustre-OST0000 complete [ 369.222916] Lustre: server umount lustre-OST0001 complete [ 377.864196] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 386.890806] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 426.242168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.457804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 426.481821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 426.590207] Lustre: lustre-MDT0000: new disk, initializing [ 426.673144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.713606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 431.658683] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 442.956966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.095241] Lustre: 15074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 443.195696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 443.199646] Lustre: Skipped 1 previous similar message [ 443.285300] Lustre: lustre-MDT0001: new disk, initializing [ 443.442483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 443.496631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 443.515552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 448.760014] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 460.143538] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 466.840248] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 474.820776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 475.206880] Lustre: lustre-OST0000: new disk, initializing [ 475.215972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 475.226282] Lustre: 17041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 475.376171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.866930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 476.888497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 477.132669] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 482.833735] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 493.432192] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 498.492820] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 498.709430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 503.821657] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 504.139376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 507.887949] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.912392] Lustre: Skipped 1 previous similar message [ 507.928301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 507.931450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.947792] Lustre: Skipped 1 previous similar message [ 512.783245] Lustre: server umount lustre-OST0000 complete [ 518.118491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.147486] Lustre: Skipped 1 previous similar message [ 518.157523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 518.171777] Lustre: Skipped 1 previous similar message [ 523.208849] Lustre: server umount lustre-MDT0000 complete [ 523.241925] LustreError: 15085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.268962] LustreError: 15085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 527.374579] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786075037 with bad export cookie 11670483874441564835 [ 527.384716] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.723863] Lustre: server umount lustre-MDT0001 complete [ 532.515219] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 23:57:21 (1786075041) [ 535.190989] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 537.042780] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 23:57:25 (1786075045) [ 544.243627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.676718] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.754746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.856827] LustreError: 19451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.426985] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 560.101080] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.119441] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 560.731043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.415744] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 574.669831] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 581.047771] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 588.569766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 595.823606] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 604.304623] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 616.422297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.434254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 616.455744] Lustre: Skipped 2 previous similar messages [ 616.479532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 616.493823] Lustre: Skipped 1 previous similar message [ 621.970430] Lustre: server umount lustre-OST0000 complete [ 631.773577] Lustre: server umount lustre-MDT0000 complete [ 631.777295] LustreError: 19446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.817297] LustreError: 19446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 635.646240] LustreError: 19432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786075146 with bad export cookie 11670483874441565724 [ 635.654548] LustreError: 19432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 635.657066] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.920892] Lustre: server umount lustre-MDT0001 complete [ 651.270882] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 654.752781] Key type lgssc unregistered [ 655.041888] LNet: 22985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.053238] LNetError: 22985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.079537] LNet: Removed LNI 192.168.202.127@tcp [ 656.172184] Key type .llcrypt unregistered [ 656.174829] Key type ._llcrypt unregistered [ 682.980681] Key type ._llcrypt registered [ 682.983293] Key type .llcrypt registered [ 683.093491] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 684.633192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 684.858391] alg: No test for adler32 (adler32-zlib) [ 686.076954] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 686.397611] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 688.151308] Key type lgssc registered [ 689.705138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 699.166741] loop: module loaded [ 699.288570] loop0: detected capacity change from 0 to 204800000 [ 699.483134] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 699.499427] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 699.931224] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.259857] loop0: detected capacity change from 0 to 204800000 [ 707.389654] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.397789] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.416182] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.640111] loop0: detected capacity change from 0 to 204800000 [ 718.650773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 718.691108] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.118381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 720.224891] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 720.302351] Lustre: lustre-MDT0000: new disk, initializing [ 720.398416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.436211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 726.391732] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 735.957487] loop1: detected capacity change from 0 to 204800000 [ 742.731868] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 743.094804] Lustre: lustre-OST07c6: new disk, initializing [ 743.100845] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 743.104240] Lustre: Skipped 1 previous similar message [ 743.108949] Lustre: 26177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 743.200501] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 751.812283] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 753.391286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 753.410058] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 753.464484] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 756.944659] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 756.954729] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 756.961115] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 756.969627] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 756.994106] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 757.007298] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 770.082165] Lustre: 26185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 770.091105] Lustre: 26185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 770.099812] Lustre: 26185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 770.104664] Lustre: 26185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 770.109500] Lustre: 26185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 770.114458] Lustre: 26185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 774.814937] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 774.826501] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 774.833098] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 774.843708] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.851112] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 774.860704] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.870877] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 774.879825] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.887473] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 774.895850] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.908696] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 774.913940] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.143404] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 778.156286] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 778.166380] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 778.176889] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.183272] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 778.195972] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.207447] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 778.218982] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.227898] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 778.236121] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.247940] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 778.254647] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.999719] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 783.011353] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 783.020741] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 783.032396] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.051359] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 783.068597] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.082798] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 783.107144] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.113040] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 783.137149] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.147698] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 783.161618] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.713056] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 794.722595] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 794.735314] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 794.744623] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 794.750575] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 794.756539] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 794.762520] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 794.779347] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 794.793260] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 794.806919] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 794.821125] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.831901] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 843.754384] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.787026] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 849.099580] Lustre: server umount lustre-OST07c6 complete [ 860.608925] Lustre: server umount lustre-MDT0000 complete [ 877.618314] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 880.250990] Key type lgssc unregistered [ 880.615490] LNet: 28744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.622667] LNetError: 28744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.650867] LNet: Removed LNI 192.168.202.127@tcp [ 881.710654] Key type .llcrypt unregistered [ 881.713164] Key type ._llcrypt unregistered [ 900.230866] Key type ._llcrypt registered [ 900.237882] Key type .llcrypt registered [ 900.384738] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 901.367506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 901.394426] alg: No test for adler32 (adler32-zlib) [ 902.467826] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 902.628671] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 904.311196] Key type lgssc registered [ 905.410780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 913.130942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 913.166717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.819368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 919.391416] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 925.200403] LustreError: 30018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 925.235767] LustreError: 30018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 927.976797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.292642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 933.452913] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 942.012121] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 949.143471] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 957.159692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 957.732027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.819447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 964.713749] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 972.463826] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 988.649096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.653758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 988.672883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.683848] Lustre: Skipped 1 previous similar message [ 993.776354] Lustre: server umount lustre-OST0000 complete [ 998.885413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.912067] Lustre: Skipped 1 previous similar message [ 998.917695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 998.928933] Lustre: Skipped 1 previous similar message [ 1004.000895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.297973] Lustre: server umount lustre-MDT0000 complete [ 1008.536566] LustreError: 29999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786075518 with bad export cookie 1853666982677549732 [ 1008.551242] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1009.177663] Lustre: server umount lustre-MDT0001 complete [ 1021.638613] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1024.467772] Key type lgssc unregistered [ 1024.692161] LNet: 33761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.698495] LNetError: 33761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.714928] LNet: Removed LNI 192.168.202.127@tcp [ 1025.593091] Key type .llcrypt unregistered [ 1025.596644] Key type ._llcrypt unregistered [ 1046.015686] Key type ._llcrypt registered [ 1046.017788] Key type .llcrypt registered [ 1046.103902] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1047.506520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1047.547603] alg: No test for adler32 (adler32-zlib) [ 1048.703787] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 1048.947376] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1050.615281] Key type lgssc registered [ 1051.927877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1061.814783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1061.833675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.390236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.203287] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1072.609715] LustreError: 35032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1072.642506] LustreError: 35032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1077.742352] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.879372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.386562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1084.602072] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1092.150588] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1098.172722] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1105.359615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1105.767643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.864566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1112.669339] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1121.221777] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1138.657375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1138.677261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.698167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.681695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.714091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.722760] Lustre: Skipped 1 previous similar message [ 1142.940753] Lustre: server umount lustre-OST0000 complete [ 1149.928864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.974070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1150.003591] Lustre: Skipped 1 previous similar message [ 1154.114975] Lustre: server umount lustre-MDT0000 complete [ 1155.054778] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1155.078908] LustreError: 35031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1159.134915] LustreError: 35013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786075669 with bad export cookie 16588524794923671556 [ 1159.160571] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.663191] Lustre: server umount lustre-MDT0001 complete [ 1173.840864] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1176.307342] Key type lgssc unregistered [ 1176.621059] LNet: 38583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1176.635378] LNetError: 38583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1176.665245] LNet: Removed LNI 192.168.202.127@tcp [ 1177.405086] Key type .llcrypt unregistered [ 1177.407171] Key type ._llcrypt unregistered [ 1204.823669] Key type ._llcrypt registered [ 1204.827576] Key type .llcrypt registered [ 1205.036354] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1206.336166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.536862] alg: No test for adler32 (adler32-zlib) [ 1207.895634] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 1208.192760] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1209.847626] Key type lgssc registered [ 1211.201908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1219.709970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1219.746357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.415426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1225.610824] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1230.819558] LustreError: 40006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.847246] LustreError: 40006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1234.896958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.248868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1240.318439] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1248.547716] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1255.155598] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1262.889697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1263.410906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.482843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1270.765672] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1279.362761] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1286.269245] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1286.624274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1286.630493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.213059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.214325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.228102] Lustre: Skipped 1 previous similar message [ 1291.369888] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1291.385431] Lustre: Skipped 2 previous similar messages [ 1291.748775] Lustre: server umount lustre-MDT0000 complete [ 1295.340213] LustreError: 41486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.365366] LustreError: 41486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1296.434655] LustreError: 42582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786075806 with bad export cookie 4506975514770773463 [ 1296.435721] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1296.448895] LustreError: 42582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1296.497804] LustreError: 39999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.525530] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [ 1300.450023] LustreError: 40001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.454872] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.482421] Lustre: Skipped 1 previous similar message [ 1301.474147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1301.476894] Lustre: Skipped 1 previous similar message [ 1303.074744] Lustre: server umount lustre-MDT0001 complete [ 1324.583825] Lustre: server umount lustre-OST0000 complete [ 1343.654281] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1347.211890] Key type lgssc unregistered [ 1347.531628] LNet: 43866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.544353] LNetError: 43866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.567542] LNet: Removed LNI 192.168.202.127@tcp [ 1348.844204] Key type .llcrypt unregistered [ 1348.846641] Key type ._llcrypt unregistered [ 1376.392809] Key type ._llcrypt registered [ 1376.398755] Key type .llcrypt registered [ 1376.597347] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1378.160001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1378.223348] alg: No test for adler32 (adler32-zlib) [ 1379.357611] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 1379.664958] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1381.383210] Key type lgssc registered [ 1382.693177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1391.880077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1391.890185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.598173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.510671] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1402.923529] LustreError: 45284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1402.946754] LustreError: 45284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1407.977992] LustreError: 45283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1409.684736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.035409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1415.490362] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1423.658610] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1430.095482] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1437.765416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1438.212235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.650026] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1453.705328] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1462.812663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1472.993505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.004634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1473.012038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1473.022853] Lustre: Skipped 1 previous similar message [ 1473.078637] Lustre: Skipped 1 previous similar message [ 1475.729652] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1477.324495] Lustre: server umount lustre-OST0000 complete [ 1497.056401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1497.065954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.087027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1498.600028] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1503.713949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1503.716039] Lustre: Skipped 2 previous similar messages [ 1508.833406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.838222] Lustre: Skipped 1 previous similar message [ 1511.392504] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1511.689124] Lustre: server umount lustre-MDT0000 complete [ 1512.932517] LustreError: 45283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.974416] LustreError: 45283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1515.712845] LustreError: 45264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076026 with bad export cookie 12394894016097186592 [ 1515.722572] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.337335] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1530.653693] Lustre: server umount lustre-MDT0001 complete [ 1541.858328] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1544.501384] Key type lgssc unregistered [ 1544.724166] LNet: 49021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.737206] LNetError: 49021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.763563] LNet: Removed LNI 192.168.202.127@tcp [ 1545.533790] Key type .llcrypt unregistered [ 1545.536167] Key type ._llcrypt unregistered [ 1566.801717] Key type ._llcrypt registered [ 1566.803568] Key type .llcrypt registered [ 1566.883522] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1568.168587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1568.181248] alg: No test for adler32 (adler32-zlib) [ 1569.253563] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 1569.496388] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1571.202086] Key type lgssc registered [ 1572.395721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1579.800207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1579.822222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.438839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.248183] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1590.762296] LustreError: 50416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1590.777175] LustreError: 50416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1594.846602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.203733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1599.482612] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1607.801700] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1614.494263] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1623.060985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1623.609031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1625.670702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1631.665731] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1640.998810] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1646.858345] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1649.993813] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1662.224855] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-07 0h16m10s [ 1664.296086] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1667.041216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1667.047969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.066254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.132795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.162731] Lustre: Skipped 4 previous similar messages [ 1669.806984] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1669.815868] Lustre: Skipped 1 previous similar message [ 1672.965512] Lustre: server umount lustre-MDT0000 complete [ 1674.222973] LustreError: 51460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1674.251345] LustreError: 51460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1677.106035] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1680.372158] LustreError: 50416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.404958] LustreError: 50416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1683.631934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1683.737302] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.968156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.070224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1684.982593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1687.543301] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1690.803785] Lustre: DEBUG MARKER: Wait for df (36774) ... [ 1692.244303] Lustre: DEBUG MARKER: done [ 1699.298979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1699.309063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.324915] Lustre: Skipped 2 previous similar messages [ 1699.333678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1700.327718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.025619] Lustre: server umount lustre-OST0000 complete [ 1710.560812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1710.564270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.571154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1710.578604] Lustre: Skipped 1 previous similar message [ 1710.595470] Lustre: Skipped 3 previous similar messages [ 1713.759143] Lustre: server umount lustre-MDT0000 complete [ 1717.751287] LustreError: 51460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1717.764050] LustreError: 51460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1717.885602] LustreError: 53041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076228 with bad export cookie 13222132241518296493 [ 1717.886601] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1717.891969] LustreError: 53041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1718.301278] Lustre: server umount lustre-MDT0001 complete [ 1731.967757] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1735.364787] Key type lgssc unregistered [ 1735.922830] LNet: 55336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.941410] LNetError: 55336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.997879] LNet: Removed LNI 192.168.202.127@tcp [ 1738.030930] Key type .llcrypt unregistered [ 1738.037402] Key type ._llcrypt unregistered [ 1782.639817] Key type ._llcrypt registered [ 1782.641755] Key type .llcrypt registered [ 1782.725360] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1784.141248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1784.188122] alg: No test for adler32 (adler32-zlib) [ 1785.369839] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 1785.656907] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1787.327182] Key type lgssc registered [ 1789.141401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1799.758521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1799.790548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.234252] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1801.311706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1801.570336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.640838] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1818.665468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.769715] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1818.799703] Lustre: 57547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1818.980571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1823.114840] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1831.464540] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1838.604143] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1848.739523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1849.110502] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1849.531922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.231747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1858.236828] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1867.872681] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1874.821945] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-07 0h19m43s [ 1876.455465] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1884.264135] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1887.042892] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1887.096248] LustreError: 57554:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1907.378353] Lustre: lustre-MDT0000: Client 48e48e37-d1b4-413a-9c05-9376ec2aeb81 (at 192.168.202.27@tcp) reconnecting [ 1927.158582] LustreError: 57554:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1930.911446] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1934.776639] Lustre: DEBUG MARKER: done [ 1947.671230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1947.672757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.682995] LustreError: Skipped 1 previous similar message [ 1947.693227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1947.701703] Lustre: Skipped 1 previous similar message [ 1947.721074] Lustre: Skipped 1 previous similar message [ 1950.709963] Lustre: server umount lustre-OST0000 complete [ 1957.359173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1957.371760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.399481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.433963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.436283] Lustre: Skipped 1 previous similar message [ 1961.041324] Lustre: server umount lustre-MDT0000 complete [ 1965.475091] LustreError: 57539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076475 with bad export cookie 81122020911165105 [ 1965.482481] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1965.487423] LustreError: 57539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1965.919275] Lustre: server umount lustre-MDT0001 complete [ 1977.665375] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1980.219945] Key type lgssc unregistered [ 1980.505084] LNet: 61880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.515577] LNetError: 61880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.532417] LNet: Removed LNI 192.168.202.127@tcp [ 1981.337571] Key type .llcrypt unregistered [ 1981.339826] Key type ._llcrypt unregistered [ 2032.368167] Key type ._llcrypt registered [ 2032.371326] Key type .llcrypt registered [ 2032.473859] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2034.225826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2034.317172] alg: No test for adler32 (adler32-zlib) [ 2035.533327] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2035.771714] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2037.463300] Key type lgssc registered [ 2039.028798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2042.895970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2042.940200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.383237] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2044.423051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2044.753483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.944863] Lustre: Failing over lustre-MDT0000 [ 2046.317964] Lustre: server umount lustre-MDT0000 complete [ 2053.458430] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:22:41 (1786076561) [ 2060.912853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.525115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.502625] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2077.502769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.733829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2077.818516] Lustre: 64692:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2078.274149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2083.537149] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2092.706698] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2099.066952] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2106.762265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2107.200677] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2107.529947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.796374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2116.029598] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2125.448826] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2131.839218] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2139.624500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2139.635312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.644428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2141.665951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2141.671060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.678582] Lustre: Skipped 1 previous similar message [ 2141.710137] Lustre: Skipped 1 previous similar message [ 2145.267834] Lustre: server umount lustre-MDT0000 complete [ 2149.661057] LustreError: 64683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076660 with bad export cookie 16983429463643895900 [ 2149.665569] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.670042] LustreError: 64683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2150.005231] Lustre: server umount lustre-MDT0001 complete [ 2154.625681] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2159.978384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2168.300232] Lustre: 63610:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786076662/real 1786076662] req@00000000d2a8280b x1872837000341504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786076678 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2168.339778] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.675610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.423174] Lustre: 63610:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786076665/real 1786076665] req@00000000ae4e8477 x1872837000341632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786076681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2173.407433] Lustre: 63613:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786076667/real 1786076667] req@00000000ab89d1d0 x1872837000341760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786076683 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2175.900674] LustreError: 66541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.921588] LustreError: 66541:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2176.009782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2176.136981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2176.992100] Lustre: 63613:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786076670/real 1786076670] req@00000000fb304ddc x1872837000342016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786076686 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2178.025983] LustreError: 66542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.119355] LustreError: 68465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.156862] LustreError: 68465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2181.153649] Lustre: 63610:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786076675/real 1786076675] req@000000002731b168 x1872837000342144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786076691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2181.157627] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2184.160701] LustreError: 68462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2184.182123] LustreError: 68462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2189.285334] LustreError: 68465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.322581] LustreError: 68465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2192.109937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2192.553293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.687717] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2197.653945] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2207.786358] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2214.980634] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2222.561198] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.564339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.581839] Lustre: Skipped 1 previous similar message [ 2222.595723] Lustre: Skipped 3 previous similar messages [ 2227.680891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.689684] Lustre: Skipped 1 previous similar message [ 2228.655759] Lustre: server umount lustre-MDT0000 complete [ 2232.667487] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076743 with bad export cookie 16983429463643898175 [ 2232.670893] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2232.677573] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2232.801855] LustreError: 68465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.805102] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2232.821558] Lustre: Skipped 1 previous similar message [ 2232.830539] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2232.835530] Lustre: Skipped 1 previous similar message [ 2233.098217] Lustre: server umount lustre-MDT0001 complete [ 2239.533258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2250.109330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.464513] LustreError: 63609:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000011d01f98 x1872837000371328/t0(0) o250->MGC192.168.202.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2258.509195] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.525990] Lustre: MGS: Client b3233258-2206-4367-a4e7-ef9d105859ff (at 0@lo) reconnecting [ 2258.538673] Lustre: MGC192.168.202.127@tcp: Connection restored to 0@lo (at 0@lo) [ 2258.559076] Lustre: Skipped 1 previous similar message [ 2258.823987] LustreError: 69200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.914918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2258.966896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2263.016556] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2264.361909] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2273.610251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2279.310101] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2279.406731] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2288.640626] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2296.591806] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2309.604795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2309.619570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.652710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.786808] Lustre: server umount lustre-MDT0000 complete [ 2315.233674] LustreError: 71151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.258755] LustreError: 71151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2316.418325] LustreError: 65426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786076826 with bad export cookie 16983429463643899554 [ 2316.420026] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.429684] LustreError: 65426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2316.825882] Lustre: server umount lustre-MDT0001 complete [ 2324.031605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2327.882908] Lustre: DEBUG MARKER: files compared the same [ 2340.812824] Lustre: server umount lustre-OST0000 complete [ 2360.708271] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2363.688168] Key type lgssc unregistered [ 2364.395650] LNet: 74427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.409194] LNetError: 74427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2364.465190] LNet: Removed LNI 192.168.202.127@tcp [ 2366.337106] Key type .llcrypt unregistered [ 2366.339374] Key type ._llcrypt unregistered [ 2397.110860] Key type ._llcrypt registered [ 2397.113960] Key type .llcrypt registered [ 2397.230735] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2398.524119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2398.599814] alg: No test for adler32 (adler32-zlib) [ 2399.975441] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2400.299565] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2401.991212] Key type lgssc registered [ 2403.521755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2414.785191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2414.815220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2416.508796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.167558] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug +malloc all [ 2425.826164] LustreError: 75845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.866486] LustreError: 75845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2430.945025] LustreError: 75846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.106973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.566988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2440.821852] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug +malloc all [ 2450.124448] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2456.823987] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2464.896150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2465.312448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2472.434571] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug +malloc all [ 2482.685770] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2485.808825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2499.039883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2499.044690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2499.056946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2501.088562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2501.091698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2501.111381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.149319] Lustre: Skipped 1 previous similar message [ 2502.812983] Lustre: server umount lustre-OST0000 complete [ 2511.331684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2511.339175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.352808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2511.372890] Lustre: Skipped 1 previous similar message [ 2511.411114] Lustre: Skipped 1 previous similar message [ 2513.721738] Lustre: server umount lustre-MDT0000 complete [ 2516.448526] LustreError: 75845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.470067] LustreError: 75845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2518.424695] LustreError: 75826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786077028 with bad export cookie 6943600887101484169 [ 2518.449261] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2518.805488] Lustre: server umount lustre-MDT0001 complete [ 2531.679651] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2534.642676] Key type lgssc unregistered [ 2534.970952] LNet: 79404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.982481] LNetError: 79404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.003874] LNet: Removed LNI 192.168.202.127@tcp [ 2536.486237] Key type .llcrypt unregistered [ 2536.493305] Key type ._llcrypt unregistered [ 2566.096526] Key type ._llcrypt registered [ 2566.104137] Key type .llcrypt registered [ 2566.235139] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2567.725927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2567.746930] alg: No test for adler32 (adler32-zlib) [ 2568.975202] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2569.379581] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2571.056621] Key type lgssc registered [ 2572.050231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2582.105641] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2582.125314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2597.727396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2602.847360] LustreError: 80797:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 2629.535246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.262366] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2645.577196] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2655.838181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.111651] Lustre: *** cfs_fail_loc=706, val=0*** [ 2656.821688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2656.959434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2661.863755] LustreError: 80822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2661.885774] LustreError: 80822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2662.204298] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2663.916875] LustreError: 81883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2663.946502] LustreError: 81883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2669.034428] LustreError: 80823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.064192] LustreError: 80823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2673.308377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.753830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2679.282388] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2688.063374] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2695.008313] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2699.245940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.274060] Lustre: Skipped 1 previous similar message [ 2699.274992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2699.277743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2704.368829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2704.387598] Lustre: Skipped 1 previous similar message [ 2704.847244] Lustre: server umount lustre-OST0000 complete [ 2709.475191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.483759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.505575] Lustre: Skipped 1 previous similar message [ 2709.521034] Lustre: Skipped 1 previous similar message [ 2714.592690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2714.606124] Lustre: Skipped 2 previous similar messages [ 2715.448863] Lustre: server umount lustre-MDT0000 complete [ 2719.342161] LustreError: 80804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786077229 with bad export cookie 7095835156826886895 [ 2719.348036] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.361371] LustreError: 80804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2719.796921] Lustre: server umount lustre-MDT0001 complete [ 2731.456945] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2734.819077] Key type lgssc unregistered [ 2735.329071] LNet: 84413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2735.345884] LNetError: 84413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2735.365403] LNet: Removed LNI 192.168.202.127@tcp [ 2736.320260] Key type .llcrypt unregistered [ 2736.323524] Key type ._llcrypt unregistered [ 2761.020554] Key type ._llcrypt registered [ 2761.023992] Key type .llcrypt registered [ 2761.157926] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2762.609573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2762.748694] alg: No test for adler32 (adler32-zlib) [ 2763.840207] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 2764.092583] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2765.791242] Key type lgssc registered [ 2767.163391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2776.344409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2776.364310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.818157] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2791.480068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.084296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2796.762160] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2797.537051] LustreError: 86518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2797.568305] LustreError: 86518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2801.648280] LustreError: 86522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.660987] LustreError: 86522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2805.539051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2805.844080] LustreError: 86518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.861416] LustreError: 86518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2805.909159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.367169] LustreError: 86521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2811.391790] LustreError: 86521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2813.519868] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2816.487741] LustreError: 86516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.512307] LustreError: 86516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2823.639562] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2823.659331] Lustre: MGS: Client 5a7bf697-6e5b-4295-b1c2-be4a41a0eae3 (at 0@lo) reconnecting [ 2823.683035] LustreError: 86532:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2823.910455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.054768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2829.554316] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2849.072625] Lustre: server umount lustre-OST0000 complete [ 2854.883176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2854.893733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.897890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.926510] Lustre: Skipped 1 previous similar message [ 2860.008473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.022950] Lustre: Skipped 2 previous similar messages [ 2865.125975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2865.147185] Lustre: Skipped 1 previous similar message [ 2865.170847] Lustre: server umount lustre-MDT0000 complete [ 2866.146578] Lustre: 85190:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786077360/real 1786077360] req@000000009ad5896d x1872837763689472/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786077376 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2866.147082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2866.177041] Lustre: 85190:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2866.216130] LustreError: 86521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2866.237911] LustreError: 86521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2870.058787] Lustre: server umount lustre-MDT0001 complete [ 2873.733267] Lustre: server umount MGS complete [ 2886.842870] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:36:34 (1786077394) [ 2920.209987] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 2930.324200] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2983.203362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.595849] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2999.699547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.165834] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3000.225331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3000.356480] Lustre: lustre-MDT0001: new disk, initializing [ 3000.474316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3000.516670] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3000.524292] Lustre: 93710:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3000.544696] LustreError: 93710:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3006.648209] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3018.888122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3019.188197] Lustre: lustre-OST0000: new disk, initializing [ 3019.192577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3019.200247] Lustre: 94477:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3019.304742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3026.533853] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3026.978494] Lustre: 94791:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000005d25235b x1872837763704320/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786077553 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3038.135668] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3046.451191] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3046.469858] Lustre: MGS: Client f5d1dc46-d4c0-41c1-8954-2ef69ccee60f (at 0@lo) reconnecting [ 3046.503530] LustreError: 93696:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3046.580991] Lustre: 92979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3046.634747] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3046.758947] Lustre: lustre-MDT0000: new disk, initializing [ 3046.947263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.974557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3046.990773] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3048.034118] Lustre: 95546:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000008b1273cb x1872837763711360/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786077574 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3052.998802] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3053.107778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3053.125202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3053.261753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3080.161804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3080.174054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.186432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3083.767047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.175345] Lustre: server umount lustre-OST0000 complete [ 3093.984305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.988125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3094.008521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3094.014190] Lustre: Skipped 3 previous similar messages [ 3098.097147] Lustre: server umount lustre-MDT0000 complete [ 3099.106504] LustreError: 93681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.130370] LustreError: 93681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3108.547875] Lustre: server umount lustre-MDT0001 complete [ 3113.227307] Lustre: server umount MGS complete [ 3127.290730] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:40:35 (1786077635) [ 3148.084963] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3151.080915] Key type lgssc unregistered [ 3151.467062] LNet: 98045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3151.481647] LNetError: 98045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3151.506503] LNet: Removed LNI 192.168.202.127@tcp [ 3152.711231] Key type .llcrypt unregistered [ 3152.712939] Key type ._llcrypt unregistered [ 3174.761761] Key type ._llcrypt registered [ 3174.766492] Key type .llcrypt registered [ 3174.919760] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3176.340426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3176.367957] alg: No test for adler32 (adler32-zlib) [ 3177.646076] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 3177.950679] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3179.634069] Key type lgssc registered [ 3181.145801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3191.551284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3191.596059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.256782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3198.753340] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3203.563940] LustreError: 99267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3203.595502] LustreError: 99267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3208.683191] LustreError: 99266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.915452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.451023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3215.984276] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3224.557668] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3232.225277] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3240.237229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3240.735382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.827580] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3258.023893] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3264.382300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3274.723994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3274.728565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3274.742063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3274.756868] Lustre: Skipped 1 previous similar message [ 3279.843415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3279.851489] Lustre: Skipped 2 previous similar messages [ 3281.020594] Lustre: server umount lustre-OST0000 complete [ 3287.521615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3287.536792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3287.562432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3290.093541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.094863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3290.143493] Lustre: Skipped 1 previous similar message [ 3291.940742] Lustre: server umount lustre-MDT0000 complete [ 3295.216623] LustreError: 101710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3295.247214] LustreError: 101710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3296.413283] LustreError: 99247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786077806 with bad export cookie 1888278464710461834 [ 3296.416941] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3296.421075] LustreError: 99247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3296.725764] Lustre: server umount lustre-MDT0001 complete [ 3311.035766] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3314.807461] Key type lgssc unregistered [ 3315.104461] LNet: 102824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3315.111900] LNetError: 102824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3315.128735] LNet: Removed LNI 192.168.202.127@tcp [ 3316.267898] Key type .llcrypt unregistered [ 3316.272227] Key type ._llcrypt unregistered [ 3335.521907] Key type ._llcrypt registered [ 3335.532038] Key type .llcrypt registered [ 3335.689779] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3337.017790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3337.185191] alg: No test for adler32 (adler32-zlib) [ 3338.368536] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 3338.595586] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3340.375246] Key type lgssc registered [ 3341.914742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3348.674715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3348.687588] LustreError: 103908:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3348.690680] LustreError: 103907:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3348.705460] LustreError: 103908:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3348.713800] LustreError: 103908:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3348.723410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.726584] LustreError: 103908:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3350.511749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3359.721223] LustreError: 103946:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.756217] LustreError: 103946:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3359.919489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3360.444495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.326506] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3372.832603] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing lsmod [ 3376.525263] LustreError: 105112:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3376.558159] LustreError: 105114:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3376.564865] LustreError: 105112:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4970 [ 3376.570422] LustreError: 105112:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3376.580601] LustreError: 105112:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3376.629983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3377.278287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.220161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3381.226164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3381.247714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.753884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.759192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.762947] Lustre: Skipped 1 previous similar message [ 3382.783093] Lustre: Skipped 1 previous similar message [ 3384.866842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3384.880703] Lustre: Skipped 3 previous similar messages [ 3387.957172] Lustre: server umount lustre-MDT0000 complete [ 3389.931710] LustreError: 103947:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3389.946721] LustreError: 103947:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3392.862238] LustreError: 103923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786077903 with bad export cookie 13585572923605026983 [ 3392.882047] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3392.896036] Lustre: 103938:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.127@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3393.073233] LustreError: 104377:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ec964e71 x1872838366614144/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3393.318432] Lustre: server umount lustre-MDT0001 complete [ 3409.041526] Lustre: server umount lustre-OST0000 complete [ 3420.107254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.579053] LustreError: 106177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3420.713047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.771050] LustreError: 106178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3428.287278] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3430.894782] LustreError: 106177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.134164] LustreError: 106178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.166250] LustreError: 106178:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3442.538749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.133384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3448.840988] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3457.793038] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3465.145977] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3472.736329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3473.131229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3480.413598] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3488.364248] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3491.571721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3502.049674] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3502.053620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.062635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3502.091201] Lustre: Skipped 1 previous similar message [ 3506.022250] Lustre: server umount lustre-OST0000 complete [ 3512.291260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3512.299132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3512.310344] Lustre: Skipped 2 previous similar messages [ 3516.697932] Lustre: server umount lustre-MDT0000 complete [ 3517.410469] LustreError: 106172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3517.438522] LustreError: 106172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3521.339859] LustreError: 106158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786078031 with bad export cookie 13585572923605027676 [ 3521.347571] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3521.351447] LustreError: 106158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3521.612923] Lustre: server umount lustre-MDT0001 complete [ 3535.420793] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3538.595726] Key type lgssc unregistered [ 3538.880055] LNet: 109715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3538.913584] LNetError: 109715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3538.948538] LNet: Removed LNI 192.168.202.127@tcp [ 3539.980906] Key type .llcrypt unregistered [ 3539.988527] Key type ._llcrypt unregistered [ 3567.217422] Key type ._llcrypt registered [ 3567.220184] Key type .llcrypt registered [ 3567.336105] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3568.464757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3568.602067] alg: No test for adler32 (adler32-zlib) [ 3569.793747] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 3570.132874] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3571.839311] Key type lgssc registered [ 3573.064844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3582.900323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3582.962136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3584.872859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3590.530768] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3595.234797] LustreError: 111134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3595.256946] LustreError: 111134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3600.356538] LustreError: 111133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3601.552074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.102648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3607.050403] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3617.770423] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3628.258471] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3637.530651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3637.735656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3645.843731] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3654.481719] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3655.166140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3660.618684] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3664.416374] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3670.496824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3670.503935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3670.508624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3670.520602] Lustre: Skipped 1 previous similar message [ 3675.622459] LustreError: 112953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3675.643987] LustreError: 112953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3675.785327] Lustre: server umount lustre-OST0000 complete [ 3680.748212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.768406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.773410] Lustre: Skipped 1 previous similar message [ 3685.857678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3685.864874] Lustre: Skipped 1 previous similar message [ 3686.759354] Lustre: server umount lustre-MDT0000 complete [ 3690.980312] LustreError: 111130:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3691.020417] LustreError: 111130:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3691.901568] LustreError: 111115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786078202 with bad export cookie 9871080922227865751 [ 3691.904579] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3691.911814] LustreError: 111115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3692.297379] Lustre: server umount lustre-MDT0001 complete [ 3705.267645] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3708.555953] Key type lgssc unregistered [ 3708.887696] LNet: 114774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3708.895793] LNetError: 114774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3708.921784] LNet: Removed LNI 192.168.202.127@tcp [ 3709.787248] Key type .llcrypt unregistered [ 3709.793124] Key type ._llcrypt unregistered [ 3730.254898] Key type ._llcrypt registered [ 3730.257887] Key type .llcrypt registered [ 3730.352387] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3731.503470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3731.643231] alg: No test for adler32 (adler32-zlib) [ 3733.013689] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 3733.374437] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3735.095322] Key type lgssc registered [ 3736.615410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3746.003684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3746.024159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3747.473656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.655638] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3757.027723] LustreError: 115996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3757.057840] LustreError: 115996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3762.157176] LustreError: 115997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3763.570880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3763.978093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3769.813632] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3778.732780] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3784.918597] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3793.434508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3793.905500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.303547] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3809.624634] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3817.482194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3829.742916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3829.754459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.770276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.801823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.809048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.833076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3832.844519] Lustre: Skipped 1 previous similar message [ 3834.175341] Lustre: server umount lustre-OST0000 complete [ 3840.996977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3841.008369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.039861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.043372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.046469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3845.190128] Lustre: server umount lustre-MDT0000 complete [ 3848.160957] LustreError: 115992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.187310] LustreError: 115992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3849.536874] LustreError: 115979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786078359 with bad export cookie 15279560904527676629 [ 3849.539428] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3849.556063] LustreError: 115979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3850.110420] Lustre: server umount lustre-MDT0001 complete [ 3861.683317] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3864.366913] Key type lgssc unregistered [ 3864.685128] LNet: 119557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3864.696549] LNetError: 119557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3864.714300] LNet: Removed LNI 192.168.202.127@tcp [ 3865.502434] Key type .llcrypt unregistered [ 3865.504538] Key type ._llcrypt unregistered [ 3889.188380] Key type ._llcrypt registered [ 3889.190927] Key type .llcrypt registered [ 3889.283847] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3890.420317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3890.447939] alg: No test for adler32 (adler32-zlib) [ 3891.565277] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 3891.841572] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3893.567302] Key type lgssc registered [ 3894.637616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3903.026321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3903.045485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3904.478524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3909.001575] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3913.710051] LustreError: 120978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3913.744126] LustreError: 120978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3918.163905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3918.578255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3923.460632] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3931.073602] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3936.693415] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3943.233664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3943.345783] LustreError: 122798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3943.530937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.670283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3950.179819] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3957.205544] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3965.119618] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3968.670539] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3968.675944] Lustre: Skipped 2 previous similar messages [ 3973.524569] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3973.532842] Lustre: Skipped 2 previous similar messages [ 3974.188955] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3974.219483] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3981.420330] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3981.441824] Lustre: Skipped 2 previous similar messages [ 3982.854452] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.27@tcp 192.168.0.[2,10]@tcp [ 3989.262397] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.27/24@tcp in log lustre-MDT0000 [ 3989.275500] Lustre: Skipped 2 previous similar messages [ 3997.725370] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.27/24@tcp [ 3997.730535] Lustre: Skipped 1 previous similar message [ 4005.355195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.366113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4005.380275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4005.399127] Lustre: Skipped 1 previous similar message [ 4006.810782] Lustre: server umount lustre-OST0000 complete [ 4015.584533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.614263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4015.623801] Lustre: Skipped 2 previous similar messages [ 4017.933742] Lustre: server umount lustre-MDT0000 complete [ 4020.705587] LustreError: 121464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4020.733080] LustreError: 121464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4022.453363] LustreError: 121713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786078532 with bad export cookie 8545241735504716958 [ 4022.461846] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4022.464195] LustreError: 121713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4022.825025] Lustre: server umount lustre-MDT0001 complete [ 4037.181346] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 4040.163364] Key type lgssc unregistered [ 4040.481591] LNet: 125767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4040.492173] LNetError: 125767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4040.508907] LNet: Removed LNI 192.168.202.127@tcp [ 4041.500440] Key type .llcrypt unregistered [ 4041.503879] Key type ._llcrypt unregistered [ 4072.326506] Key type ._llcrypt registered [ 4072.328968] Key type .llcrypt registered [ 4072.433106] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4073.371786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4073.409993] alg: No test for adler32 (adler32-zlib) [ 4074.811807] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 4075.058477] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 4076.751441] Key type lgssc registered [ 4077.885900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4088.438994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4088.490312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.047559] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4090.110886] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.27/24@tcp [ 4090.113291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.851834] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4100.578979] LustreError: 127327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4100.605725] LustreError: 127327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4105.711050] LustreError: 127328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.582797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4107.056988] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4107.129923] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.27/24@tcp [ 4107.205169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4111.747910] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4119.697842] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4126.077351] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4133.922839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4134.343051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4136.371962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4141.808049] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4151.727633] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4162.018919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4162.022203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.046670] LustreError: Skipped 1 previous similar message [ 4162.062703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4162.084120] Lustre: Skipped 1 previous similar message [ 4167.138424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4167.157972] Lustre: Skipped 1 previous similar message [ 4167.595952] Lustre: server umount lustre-OST0000 complete [ 4172.256628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.259884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4172.267465] Lustre: Skipped 2 previous similar messages [ 4177.393212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4177.406488] Lustre: Skipped 2 previous similar messages [ 4178.454623] Lustre: server umount lustre-MDT0000 complete [ 4182.502105] LustreError: 127324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.515452] LustreError: 127324:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4182.864039] LustreError: 127309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786078693 with bad export cookie 3455649364193761861 [ 4182.873146] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4182.883124] LustreError: 127309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4183.358557] Lustre: server umount lustre-MDT0001 complete [ 4196.661444] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 4199.835691] Key type lgssc unregistered [ 4200.209945] LNet: 130917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4200.223604] LNetError: 130917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4200.245230] LNet: Removed LNI 192.168.202.127@tcp [ 4201.193337] Key type .llcrypt unregistered [ 4201.195259] Key type ._llcrypt unregistered [ 4235.984108] Key type ._llcrypt registered [ 4235.985815] Key type .llcrypt registered [ 4236.108384] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 4252.865359] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4254.073511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4254.145837] alg: No test for adler32 (adler32-zlib) [ 4255.250444] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 4255.476439] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 4257.127197] Key type lgssc registered [ 4258.680781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4302.490366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4302.524594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.756410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4303.805773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4303.905709] Lustre: lustre-MDT0000: new disk, initializing [ 4304.072951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4304.121586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4309.920912] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4322.656413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.900473] Lustre: 134319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4323.060680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4323.069248] Lustre: Skipped 1 previous similar message [ 4323.218658] Lustre: lustre-MDT0001: new disk, initializing [ 4323.316527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4323.344986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4323.354034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4328.867712] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4340.941199] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4348.749482] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4357.432186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4357.761592] Lustre: lustre-OST0000: new disk, initializing [ 4357.771252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4357.779855] Lustre: 136299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4357.927145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4363.815421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4363.828403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4364.032371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4365.440729] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4376.577826] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4382.284199] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4382.662140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4388.362274] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4388.591773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4394.982214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4394.982662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4394.998057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4400.099891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4400.111857] Lustre: Skipped 2 previous similar messages [ 4405.220908] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4405.230103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4405.250215] Lustre: Skipped 1 previous similar message [ 4405.627487] Lustre: server umount lustre-OST0000 complete [ 4410.337769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4410.339346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4410.367469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4410.387931] Lustre: Skipped 2 previous similar messages [ 4410.425756] Lustre: Skipped 1 previous similar message [ 4415.456798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4415.461777] Lustre: Skipped 1 previous similar message [ 4416.740734] Lustre: server umount lustre-MDT0000 complete [ 4421.031834] LustreError: 135122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786078931 with bad export cookie 152065616501905564 [ 4421.035560] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4421.051466] LustreError: 135122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4421.847057] Lustre: server umount lustre-MDT0001 complete [ 4432.803264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.401559] LustreError: 138524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.437071] LustreError: 138524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4433.505404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4438.499597] LustreError: 138525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.504724] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4442.613462] LustreError: 138524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4447.716257] LustreError: 138525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4451.501702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4451.985814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4457.662566] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4466.576543] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4473.378480] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4482.250733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4482.874463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4491.761738] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4501.300909] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4507.607399] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4507.922427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4514.202833] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4514.486391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4524.597890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4524.944513] Lustre: lustre-OST0001: new disk, initializing [ 4524.949803] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4524.971961] Lustre: 141896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4525.087680] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4532.631331] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4533.896764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4533.911750] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4534.033625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4541.780597] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4542.135703] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4547.296421] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4547.599108] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4555.413578] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff997910aa6800.ost_server_uuid 50 [ 4557.274726] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff997910aa6800.ost_server_uuid in IDLE state after 0 sec [ 4570.085929] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4570.096795] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4570.112851] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4570.129927] Lustre: Skipped 1 previous similar message [ 4570.148408] Lustre: Skipped 1 previous similar message [ 4572.974325] Lustre: server umount lustre-OST0001 complete [ 4575.210325] LustreError: 140346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4575.239726] LustreError: 140346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4579.297494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4579.307072] LustreError: Skipped 1 previous similar message [ 4579.313324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.588643] Lustre: server umount lustre-OST0000 complete [ 4590.051623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4590.059738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4590.093223] Lustre: Skipped 1 previous similar message [ 4590.105567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4590.116903] Lustre: Skipped 3 previous similar messages [ 4594.617125] Lustre: server umount lustre-MDT0000 complete [ 4595.680471] LustreError: 138524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4595.711386] LustreError: 138524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4598.838812] LustreError: 138506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786079109 with bad export cookie 152065616501906446 [ 4598.847988] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4598.861960] LustreError: 138506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4599.319760] Lustre: server umount lustre-MDT0001 complete [ 4622.111672] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 4626.631699] Key type lgssc unregistered [ 4626.955390] LNet: 144900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4626.963472] LNetError: 144900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4628.019493] LNet: Removed LNI 192.168.202.127@tcp [ 4629.058410] Key type .llcrypt unregistered [ 4629.068105] Key type ._llcrypt unregistered [ 4679.580511] Key type ._llcrypt registered [ 4679.583215] Key type .llcrypt registered [ 4679.703205] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 4700.603145] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4703.013673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4703.276770] alg: No test for adler32 (adler32-zlib) [ 4704.663513] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 4705.054532] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 4706.864254] Key type lgssc registered [ 4708.867616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4752.010665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4752.041503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4753.346370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4753.382735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4753.548900] Lustre: lustre-MDT0000: new disk, initializing [ 4753.622360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.638388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4758.829489] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4770.681342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4770.912529] Lustre: 149063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4771.061948] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4771.077553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4771.087805] Lustre: Skipped 1 previous similar message [ 4771.165661] Lustre: lustre-MDT0001: new disk, initializing [ 4771.268538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4771.320400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4771.341205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4777.121294] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4788.436492] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4796.090828] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4805.042190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4805.434232] Lustre: lustre-OST0000: new disk, initializing [ 4805.447172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4805.459460] Lustre: 151035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4805.578617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4811.886392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4811.894558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4812.074217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4813.276522] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4824.322642] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4830.194321] Lustre: Failing over lustre-OST0000 [ 4830.649531] Lustre: server umount lustre-OST0000 complete [ 4848.095118] Lustre: 147285:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786079342/real 1786079342] req@0000000003066dc8 x1872839798983040/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786079358 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4848.115245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4849.122584] Lustre: 147287:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786079343/real 1786079343] req@00000000eb898d62 x1872839798983168/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786079359 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4849.123224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4849.161879] Lustre: 147287:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4851.531170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4852.047261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.059350] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4854.050441] Lustre: 147287:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786079348/real 1786079348] req@0000000086d1d3ba x1872839798985216/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786079364 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4854.086167] Lustre: 147287:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4858.234014] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4859.168090] Lustre: 147286:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786079353/real 1786079353] req@00000000000e787c x1872839798986112/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786079369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4859.346325] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4859.346851] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4860.137985] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4864.359550] Lustre: 147286:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786079358/real 1786079358] req@00000000f3a8baf9 x1872839798987136/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786079374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4864.393110] Lustre: 147286:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4865.213659] Lustre: Failing over lustre-MDT0000 [ 4865.470336] Lustre: server umount lustre-MDT0000 complete [ 4868.580440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4868.583948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4868.630119] Lustre: Skipped 2 previous similar messages [ 4873.699325] LustreError: 149076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4873.722239] LustreError: 149076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4874.355575] LustreError: 150433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4878.818190] LustreError: 151704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4878.852491] LustreError: 151704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4883.944843] LustreError: 149075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4883.972409] LustreError: 149075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4884.959275] Lustre: 147286:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786079379/real 1786079379] req@0000000049237bd7 x1872839799000448/t0(0) o400->MGC192.168.202.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786079395 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4885.009192] Lustre: 147286:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4885.030223] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4885.926489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.083509] LustreError: 149075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.104256] LustreError: 149075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4895.209172] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x2dbe40419ba2ea3 [ 4895.235501] Lustre: MGC192.168.202.127@tcp: Connection restored to 0@lo (at 0@lo) [ 4895.247411] Lustre: Skipped 1 previous similar message [ 4895.622580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.679791] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4899.975753] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4900.883517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4900.931428] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4901.050351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4902.430550] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4915.684199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4915.700511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4915.718853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4919.903880] Lustre: server umount lustre-OST0000 complete [ 4926.454335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.456877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4926.475689] Lustre: Skipped 2 previous similar messages [ 4926.502297] Lustre: Skipped 3 previous similar messages [ 4931.559123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4931.572485] Lustre: Skipped 1 previous similar message [ 4931.699343] Lustre: server umount lustre-MDT0000 complete [ 4936.621254] LustreError: 149057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786079447 with bad export cookie 206008914238123683 [ 4936.635292] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4936.641807] LustreError: 149057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4937.140462] Lustre: server umount lustre-MDT0001 complete [ 4948.947246] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 4952.747686] Key type lgssc unregistered [ 4953.044527] LNet: 154705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4953.051609] LNetError: 154705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4953.076863] LNet: Removed LNI 192.168.202.127@tcp [ 4954.464133] Key type .llcrypt unregistered [ 4954.469232] Key type ._llcrypt unregistered [ 4998.569805] Key type ._llcrypt registered [ 4998.572409] Key type .llcrypt registered [ 4998.730438] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5000.539443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5000.660092] alg: No test for adler32 (adler32-zlib) [ 5001.889869] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 5002.248264] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 5003.953076] Key type lgssc registered [ 5005.715318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5013.860957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5013.899988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5015.388980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5020.663487] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5024.739730] LustreError: 156232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5024.750408] LustreError: 156232:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5029.858129] LustreError: 156231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5032.172328] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5034.977514] LustreError: 156232:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5036.993968] LustreError: 156260:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5037.006552] LustreError: 156260:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5037.021510] LustreError: 156260:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 5037.610538] Lustre: server umount lustre-MDT0000 complete [ 5054.829456] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 5065.520095] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5109.094831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5109.509296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5109.604136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5109.831961] Lustre: lustre-MDT0000: new disk, initializing [ 5109.956408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.996791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5115.737692] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5127.900765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5128.105832] Lustre: 159565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5128.227765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5128.241206] Lustre: Skipped 1 previous similar message [ 5128.403235] Lustre: lustre-MDT0001: new disk, initializing [ 5128.517407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5128.565632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5128.597667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5134.221552] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5146.483323] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5152.581930] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5159.983780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5160.327305] Lustre: lustre-OST0000: new disk, initializing [ 5160.332443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5160.340276] Lustre: 161534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5160.470339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5167.163106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5167.178404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5167.282107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5167.934543] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5178.669700] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5183.854893] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5185.945525] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:14:54 (1786079694) [ 5428.705343] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.708788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5428.737093] Lustre: Skipped 2 previous similar messages [ 5428.750987] Lustre: Skipped 2 previous similar messages [ 5433.620855] Lustre: server umount lustre-MDT0000 complete [ 5433.828981] LustreError: 160427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5433.843916] LustreError: 160427:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5437.306705] LustreError: 160370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786079947 with bad export cookie 17693038480766318107 [ 5437.309800] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5437.312730] LustreError: 160370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5438.944312] LustreError: 161770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.946536] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.975759] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5442.976188] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5443.720664] Lustre: server umount lustre-MDT0001 complete [ 5453.820586] Lustre: server umount lustre-OST0000 complete [ 5463.240552] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 5473.594442] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5521.504663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.731908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5521.759349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5521.846494] Lustre: lustre-MDT0000: new disk, initializing [ 5521.943903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5521.968428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5528.344721] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5541.462949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.668254] Lustre: 166199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5541.755342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5541.758203] Lustre: Skipped 1 previous similar message [ 5541.827428] Lustre: lustre-MDT0001: new disk, initializing [ 5541.927057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5542.002025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5542.012133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5547.520860] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5559.221837] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5566.185506] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5575.039721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5575.511678] Lustre: lustre-OST0000: new disk, initializing [ 5575.517120] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5575.530500] Lustre: 168167:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5575.671568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5582.488476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5582.507654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5582.701985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5584.264606] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5596.162563] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5602.169930] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5602.387736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5607.798403] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5608.104460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5613.537684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.548032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5613.567206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5613.569902] Lustre: Skipped 1 previous similar message [ 5618.663547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5618.675567] Lustre: Skipped 1 previous similar message [ 5624.287168] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5624.552261] Lustre: server umount lustre-OST0000 complete [ 5634.016607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5634.020076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5634.036069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5634.056300] Lustre: Skipped 2 previous similar messages [ 5634.076455] Lustre: Skipped 3 previous similar messages [ 5635.549746] Lustre: server umount lustre-MDT0000 complete [ 5639.137734] LustreError: 166209:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.173412] LustreError: 166209:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5639.733981] LustreError: 166192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786080150 with bad export cookie 17693038480766476937 [ 5639.744690] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5639.746666] LustreError: 166192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5640.321486] Lustre: server umount lustre-MDT0001 complete [ 5651.188751] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:22:39 (1786080159) [ 5669.007577] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 5679.627818] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5725.037488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5725.391953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5725.460249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5725.597990] Lustre: lustre-MDT0000: new disk, initializing [ 5725.735058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5725.757585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5731.132495] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5743.921801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.046271] Lustre: 172928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5744.364744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5750.064079] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5760.475910] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5766.712520] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5774.557902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5774.953401] Lustre: lustre-OST0000: new disk, initializing [ 5774.956121] Lustre: Skipped 1 previous similar message [ 5774.966334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5774.971228] Lustre: Skipped 2 previous similar messages [ 5774.976916] Lustre: 174907:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5781.626168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5781.641092] Lustre: Skipped 1 previous similar message [ 5781.649958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5781.838686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5782.236966] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5794.579538] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5807.595867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5807.597158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5807.612593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5807.618145] LustreError: Skipped 1 previous similar message [ 5807.623973] Lustre: Skipped 1 previous similar message [ 5810.227929] Lustre: server umount lustre-OST0000 complete [ 5816.291189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5816.301055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5821.374251] Lustre: server umount lustre-MDT0000 complete [ 5822.952652] LustreError: 172938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.981845] LustreError: 172938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5826.558412] LustreError: 174906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786080337 with bad export cookie 17693038480766477819 [ 5826.592775] LustreError: 174906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5826.593440] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5827.089167] Lustre: server umount lustre-MDT0001 complete [ 5838.504223] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:25:46 (1786080346) [ 5864.382572] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 5879.138844] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5917.048524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.353217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5917.384799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5917.464504] Lustre: lustre-MDT0000: new disk, initializing [ 5917.559203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5917.562148] Lustre: Skipped 2 previous similar messages [ 5917.593185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5922.513338] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5934.334769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5934.459772] Lustre: 179232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5934.832872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5940.108411] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5951.625288] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5958.671131] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5967.364291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5967.865457] Lustre: 181212:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5975.143385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5975.253403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5975.635540] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5986.838512] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5996.006182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.012506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5996.014896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5996.014906] Lustre: Skipped 4 previous similar messages [ 5996.027514] Lustre: Skipped 2 previous similar messages [ 5996.063585] LustreError: Skipped 1 previous similar message [ 6001.123822] LustreError: 181191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.149052] LustreError: 181191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6001.255307] Lustre: server umount lustre-OST0000 complete [ 6020.063188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6020.384262] Lustre: server umount lustre-MDT0000 complete [ 6024.966594] LustreError: 179225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786080535 with bad export cookie 17693038480766478904 [ 6024.968902] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6024.974214] LustreError: 179225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6025.365994] Lustre: server umount lustre-MDT0001 complete [ 6038.849724] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 6042.981520] Key type lgssc unregistered [ 6043.314331] LNet: 183069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6043.322427] LNetError: 183069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6043.338442] LNet: Removed LNI 192.168.202.127@tcp [ 6044.442695] Key type .llcrypt unregistered [ 6044.445893] Key type ._llcrypt unregistered [ 6070.952577] Key type ._llcrypt registered [ 6070.954917] Key type .llcrypt registered [ 6071.133729] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 6072.428079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6072.500128] alg: No test for adler32 (adler32-zlib) [ 6073.675206] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 6073.870055] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 6075.519237] Key type lgssc registered [ 6076.677244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6087.613865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6087.648135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.724394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6094.752640] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6099.977243] LustreError: 184489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.995798] LustreError: 184489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6104.938763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6105.061236] LustreError: 184490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6105.715284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6110.273261] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6119.859733] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6127.255582] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6136.272515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6136.795224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.931214] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6152.389978] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6160.446864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6185.953512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.977466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.983638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6188.894055] Lustre: server umount lustre-OST0000 complete [ 6196.198424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.213081] Lustre: Skipped 1 previous similar message [ 6196.226544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6196.231838] Lustre: Skipped 1 previous similar message [ 6200.114038] Lustre: server umount lustre-MDT0000 complete [ 6201.318993] LustreError: 184486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6201.353691] LustreError: 184486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6204.802343] LustreError: 184468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786080715 with bad export cookie 16839629196833030910 [ 6204.816527] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6205.320814] Lustre: server umount lustre-MDT0001 complete [ 6220.843866] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 6223.508110] Key type lgssc unregistered [ 6223.766380] LNet: 188047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6223.779901] LNetError: 188047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6223.800143] LNet: Removed LNI 192.168.202.127@tcp [ 6224.867616] Key type .llcrypt unregistered [ 6224.871121] Key type ._llcrypt unregistered [ 6255.933172] Key type ._llcrypt registered [ 6255.935241] Key type .llcrypt registered [ 6256.033431] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 6257.313335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6257.356180] alg: No test for adler32 (adler32-zlib) [ 6258.499745] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 6258.827910] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 6260.607208] Key type lgssc registered [ 6261.908756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6270.676514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6270.694218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.387152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6277.510630] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6281.704381] LustreError: 189467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6281.714041] LustreError: 189467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6286.816760] LustreError: 189468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6287.401978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6287.722328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6293.191417] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6302.672293] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6310.001517] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6317.643748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6318.071334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6324.804695] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6332.541165] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6341.609565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6351.842525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.846150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6351.892622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6351.927479] Lustre: Skipped 1 previous similar message [ 6355.090881] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 6356.962294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6356.965058] Lustre: Skipped 1 previous similar message [ 6357.822671] Lustre: server umount lustre-OST0000 complete [ 6367.092816] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99790389d000.ost_server_uuid 50 [ 6368.720589] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6382.561083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6382.563916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6382.581203] Lustre: Skipped 2 previous similar messages [ 6388.709616] LustreError: 189467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6388.927525] Lustre: server umount lustre-MDT0000 complete [ 6392.783660] LustreError: 189448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786080903 with bad export cookie 1951329005412646850 [ 6392.790607] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6392.791746] LustreError: 189448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6393.035749] Lustre: server umount lustre-MDT0001 complete [ 6402.933450] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:35:11 (1786080911) [ 6410.423713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6410.837114] LustreError: 193409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6410.864879] LustreError: 193409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6410.967733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6415.885588] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6416.352166] LustreError: 193410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6424.610093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.127070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6430.162887] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6438.316235] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6444.093682] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6450.695158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6451.067447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6456.200350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6458.158574] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6465.815121] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6473.413721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6473.621224] Lustre: lustre-OST0001: new disk, initializing [ 6473.625839] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6473.631440] Lustre: 196250:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6473.737423] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6479.030612] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6481.499059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6481.512457] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6481.647125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6490.412932] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6496.224222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6496.229593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.245416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6496.252272] Lustre: Skipped 3 previous similar messages [ 6497.263654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6499.451644] Lustre: server umount lustre-OST0000 complete [ 6501.501612] LustreError: 195231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6501.521541] LustreError: 195231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6506.724490] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6506.966835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6512.327682] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6512.645599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6517.734544] LustreError: 195231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6517.757795] LustreError: 195231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6527.975836] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6527.994857] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6528.003404] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6528.008733] Lustre: Skipped 2 previous similar messages [ 6528.028710] Lustre: Skipped 4 previous similar messages [ 6533.184196] Lustre: server umount lustre-OST0001 complete [ 6537.698387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6537.722225] LustreError: Skipped 1 previous similar message [ 6537.732707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.838125] Lustre: server umount lustre-MDT0000 complete [ 6548.178265] LustreError: 193390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786081058 with bad export cookie 1951329005412647921 [ 6548.179173] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6548.186566] LustreError: 193390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6548.412942] Lustre: server umount lustre-MDT0001 complete [ 6582.330573] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:38:10 (1786081090) [ 6591.949782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6592.243686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6592.284616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6592.490617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6597.094665] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6607.901691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6608.014782] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6608.118202] Lustre: 199694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6613.347593] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6621.886741] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6628.702545] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6637.679669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.981522] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6644.810688] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6646.462905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6653.522450] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6661.087204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.166631] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6661.429420] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6661.438796] Lustre: Skipped 2 previous similar messages [ 6668.664875] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6671.546694] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6678.305616] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6682.092505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6682.113299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6682.123355] Lustre: Skipped 2 previous similar messages [ 6682.141932] Lustre: Skipped 6 previous similar messages [ 6695.391234] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6695.479652] Lustre: server umount lustre-OST0000 complete [ 6697.442074] LustreError: 201542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6697.476981] LustreError: 201542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6712.803102] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6712.805622] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6712.818663] LustreError: Skipped 1 previous similar message [ 6712.851659] Lustre: Skipped 1 previous similar message [ 6717.921317] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6717.950633] Lustre: Skipped 10 previous similar messages [ 6718.522528] Lustre: server umount lustre-OST0001 complete [ 6729.208722] Lustre: server umount lustre-MDT0000 complete [ 6733.866892] LustreError: 199688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786081244 with bad export cookie 1951329005412649328 [ 6733.868604] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6733.883022] LustreError: 199688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6734.337427] Lustre: server umount lustre-MDT0001 complete [ 6759.801596] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:41:08 (1786081268) [ 6777.667650] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 6788.540277] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 6828.134892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.414396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6828.438773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6828.507762] Lustre: lustre-MDT0000: new disk, initializing [ 6828.575398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6828.607964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6832.786731] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6843.479488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.606766] Lustre: 207798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6843.679298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6843.684072] Lustre: Skipped 1 previous similar message [ 6843.868922] Lustre: lustre-MDT0001: new disk, initializing [ 6844.042816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6844.058575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6849.594457] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6859.516966] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6865.445707] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6872.940543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.244935] Lustre: lustre-OST0000: new disk, initializing [ 6873.249199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6873.264836] Lustre: 209765:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6880.547410] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6882.999663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6883.015901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6883.149860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6893.704906] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6899.003882] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6899.359879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6904.569827] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6904.783454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6908.899209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6908.905695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6908.922632] Lustre: Skipped 2 previous similar messages [ 6908.927419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6908.934075] Lustre: Skipped 4 previous similar messages [ 6913.192759] Lustre: server umount lustre-OST0000 complete [ 6923.844538] Lustre: server umount lustre-MDT0000 complete [ 6924.257308] LustreError: 207808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.272926] LustreError: 207808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6927.553967] LustreError: 207789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786081437 with bad export cookie 1951329005412651120 [ 6927.557679] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6927.565893] LustreError: 207789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6927.886173] Lustre: server umount lustre-MDT0001 complete [ 6936.896652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6941.931698] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6951.547044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6957.218918] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6965.306106] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6972.644592] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6980.250539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6988.627648] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6996.546528] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7002.147975] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7002.578105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7007.554195] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7007.743592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7012.348744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7012.353271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7012.354492] LustreError: Skipped 2 previous similar messages [ 7012.394305] Lustre: Skipped 4 previous similar messages [ 7024.095251] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7024.470741] Lustre: server umount lustre-OST0000 complete [ 7032.863630] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7037.745203] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7067.531806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7074.483683] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7078.125608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7078.139927] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7078.154862] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7082.896924] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7085.831760] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 7093.743667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7093.766026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7093.774094] Lustre: Skipped 8 previous similar messages [ 7096.508903] Lustre: server umount lustre-OST0000 complete [ 7106.877790] Lustre: server umount lustre-MDT0000 complete [ 7109.104886] LustreError: 211985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7109.151615] LustreError: 211985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7110.724758] LustreError: 212719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786081621 with bad export cookie 1951329005412652002 [ 7110.740432] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7111.022832] Lustre: server umount lustre-MDT0001 complete [ 7120.002113] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:47:09 (1786081629) [ 7127.453957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7127.977778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7127.984906] Lustre: Skipped 6 previous similar messages [ 7133.435224] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7141.641574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7146.490591] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7154.168770] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7159.676863] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7166.644722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7174.257446] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7182.125490] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7187.248801] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7187.483784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7192.197320] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7192.541892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7200.205608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7200.455751] Lustre: lustre-OST0001: new disk, initializing [ 7200.459883] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7200.467142] Lustre: 221326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7206.720526] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7208.117918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7208.131337] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7208.231314] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7216.972297] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7221.855550] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7222.114699] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7227.068186] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7227.304937] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7234.034193] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7234.037468] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7234.043183] LustreError: Skipped 2 previous similar messages [ 7234.065558] Lustre: Skipped 5 previous similar messages [ 7235.605467] Lustre: server umount lustre-OST0001 complete [ 7244.035652] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7244.436044] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7249.570138] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7249.740496] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7278.486937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7284.216559] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7284.247940] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7284.253546] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7284.261219] Lustre: Skipped 1 previous similar message [ 7285.282031] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7293.333251] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7295.161758] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7311.327143] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7311.564585] Lustre: server umount lustre-OST0001 complete [ 7336.139709] Lustre: server umount lustre-OST0000 complete [ 7346.700597] Lustre: server umount lustre-MDT0000 complete [ 7351.663664] LustreError: 217937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786081862 with bad export cookie 1951329005412652968 [ 7351.676534] LustreError: 217937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7351.685096] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7358.122505] Lustre: server umount lustre-MDT0001 complete [ 7383.605658] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:51:31 (1786081891) [ 7392.644324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.929213] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7392.965554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7397.561513] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7407.068469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7407.172368] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7407.194710] Lustre: 226572:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7412.297914] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7421.002368] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7427.326451] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7434.809152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.199448] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7442.936613] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7451.775785] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7461.825308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7461.891071] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7468.949550] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7477.303809] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7478.847392] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7484.012637] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7484.253649] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7489.760641] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7489.983906] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7495.511975] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff997911b9f800.ost_server_uuid 50 [ 7497.069328] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff997911b9f800.ost_server_uuid in IDLE state after 0 sec [ 7498.303934] Lustre: Permanently deactivating lustre-OST0001 [ 7498.325311] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7502.263940] Lustre: Permanently reactivating lustre-OST0001 [ 7509.285618] Lustre: server umount lustre-OST0001 complete [ 7511.530373] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7511.531466] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7511.549442] LustreError: Skipped 3 previous similar messages [ 7511.549878] LustreError: 228414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7511.549886] LustreError: 228414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 7511.619338] Lustre: Skipped 7 previous similar messages [ 7514.083652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7514.093033] Lustre: Skipped 19 previous similar messages [ 7519.946724] Lustre: server umount lustre-OST0000 complete [ 7531.025357] Lustre: server umount lustre-MDT0000 complete [ 7535.345353] LustreError: 228431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082045 with bad export cookie 1951329005412654249 [ 7535.355675] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7535.362503] LustreError: 228431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7535.911132] Lustre: server umount lustre-MDT0001 complete [ 7563.913679] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:54:32 (1786082072) [ 7573.635992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.996437] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7574.031271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7574.040855] Lustre: Skipped 5 previous similar messages [ 7579.723855] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7589.284913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.366064] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7589.393185] Lustre: 233265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7596.186126] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7608.009566] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7616.799698] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7626.122645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7626.377372] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7626.399887] Lustre: Permanently deactivating lustre-OST0000 [ 7626.413790] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7633.687442] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7635.018717] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7642.288563] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7650.694987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7650.818765] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7651.144179] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7651.150410] Lustre: Skipped 11 previous similar messages [ 7658.135296] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7659.220395] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7666.322120] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7668.181387] Lustre: Permanently reactivating lustre-OST0000 [ 7668.190944] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7668.194863] Lustre: Skipped 2 previous similar messages [ 7668.974203] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7668.975407] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 34 seconds. I think it's dead, and I am evicting it. exp 00000000158869df, cur 1786082179 deadline 1786082175 last 1786082145 [ 7668.985276] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7668.988099] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7669.057101] Lustre: Skipped 1 previous similar message [ 7669.096519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7692.619372] Lustre: server umount lustre-OST0001 complete [ 7702.649504] Lustre: server umount lustre-OST0000 complete [ 7714.002628] Lustre: server umount lustre-MDT0000 complete [ 7718.961800] LustreError: 234005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082229 with bad export cookie 1951329005412655901 [ 7718.966722] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7718.982614] LustreError: 234005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7719.751712] Lustre: server umount lustre-MDT0001 complete [ 7736.422534] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 7739.724359] Key type lgssc unregistered [ 7739.887705] LNet: 238151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7739.894060] LNetError: 238151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7739.904837] LNet: Removed LNI 192.168.202.127@tcp [ 7740.826082] Key type .llcrypt unregistered [ 7740.830771] Key type ._llcrypt unregistered [ 7768.702794] Key type ._llcrypt registered [ 7768.714673] Key type .llcrypt registered [ 7768.948536] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 7770.588132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7770.753561] alg: No test for adler32 (adler32-zlib) [ 7772.076790] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 7772.484658] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 7774.193328] Key type lgssc registered [ 7775.910173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7788.735373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7788.799249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7790.590984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7795.721938] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7799.778608] LustreError: 239619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7799.809254] LustreError: 239619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7804.903526] LustreError: 239620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7806.393816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.565209] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7806.621816] Lustre: 239609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7806.625837] Lustre: Permanently deactivating lustre-MDT0001 [ 7806.640774] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7806.841324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7812.061577] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7815.730442] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7821.501626] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7829.052541] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7836.483829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7836.884826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7846.200133] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7852.293364] LustreError: 241462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7852.345134] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7857.667773] LustreError: 242005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7857.759558] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7859.763706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7864.823117] LustreError: 241461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7864.860807] LustreError: 241461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7865.953611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7866.309756] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7871.497681] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7871.506641] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7873.149419] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7882.408709] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7886.377918] Lustre: Permanently reactivating lustre-MDT0001 [ 7886.407677] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7886.417264] Lustre: Skipped 2 previous similar messages [ 7889.298543] Lustre: lustre-MDT0001: Client 83054b0d-9c89-4604-a38c-2b1d89859155 (at 192.168.202.27@tcp) reconnecting [ 7895.143780] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7895.158140] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7895.169355] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7895.176530] LustreError: 243335:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7895.180696] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7895.184333] LustreError: 243335:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7895.184458] LustreError: 243335:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.127@tcp: cfg command failed: rc = -108 [ 7895.186228] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7895.186228] [ 7895.192544] LustreError: 239611:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7897.016758] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7899.065636] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7905.827874] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7906.111326] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7911.511742] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7911.731751] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7914.651611] Lustre: Permanently deactivating lustre-MDT0001 [ 7914.685078] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7914.698518] Lustre: Skipped 2 previous similar messages [ 7922.860193] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7929.828749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7929.841188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7929.861989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7931.045627] Lustre: server umount lustre-MDT0000 complete [ 7932.896581] LustreError: 240366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7932.900579] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7932.917109] LustreError: 240366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7932.941809] Lustre: Skipped 1 previous similar message [ 7935.459297] LustreError: 239599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082445 with bad export cookie 16074346118294961092 [ 7935.470853] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7935.473553] LustreError: 239599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7935.960816] Lustre: server umount lustre-MDT0001 complete [ 7942.595927] Lustre: server umount lustre-OST0000 complete [ 7955.291665] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:01:04 (1786082464) [ 7965.626456] Lustre: server umount lustre-OST0001 complete [ 7972.276732] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 7981.529949] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8017.035260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8017.261528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8017.272803] Lustre: Skipped 2 previous similar messages [ 8017.301088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8017.368506] Lustre: lustre-MDT0000: new disk, initializing [ 8017.447401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.466484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8021.800459] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8032.933923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8033.218922] Lustre: 247894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8033.309436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8033.318039] Lustre: Skipped 1 previous similar message [ 8033.443754] Lustre: lustre-MDT0001: new disk, initializing [ 8033.568978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8033.648267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8033.664793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8038.806395] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8049.279297] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8055.245329] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8061.978241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8062.197695] Lustre: lustre-OST0000: new disk, initializing [ 8062.201389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8062.206610] Lustre: 249863:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8062.297291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.640516] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8072.328474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8072.351448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8072.441124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8078.113298] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8080.461440] LustreError: 247901:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8089.079903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8089.270408] Lustre: lustre-OST0001: new disk, initializing [ 8089.274333] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8089.281213] Lustre: 250976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8090.559112] LustreError: 247901:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8090.591987] LustreError: 247900:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8095.863212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8095.880481] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8095.992233] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8096.436778] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8100.664099] LustreError: 247900:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8105.930517] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8107.673686] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8111.591195] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8111.591762] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8111.605622] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8111.646394] Lustre: Skipped 2 previous similar messages [ 8115.944724] Lustre: server umount lustre-OST0001 complete [ 8116.709795] LustreError: 249848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8116.725822] LustreError: 249848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8134.113079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8134.126056] LustreError: Skipped 1 previous similar message [ 8134.126809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8134.136092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8134.146534] Lustre: Skipped 1 previous similar message [ 8134.158552] Lustre: Skipped 2 previous similar messages [ 8137.196241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8137.202493] Lustre: Skipped 1 previous similar message [ 8138.900980] Lustre: server umount lustre-OST0000 complete [ 8146.400326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8146.422425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8146.446594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8149.021848] Lustre: server umount lustre-MDT0000 complete [ 8152.545585] LustreError: 247904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8152.560642] LustreError: 247904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8153.249291] LustreError: 247884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082663 with bad export cookie 16074346118294963899 [ 8153.266361] LustreError: 247884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8153.272850] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8160.264357] Lustre: server umount lustre-MDT0001 complete [ 8172.754705] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8176.835855] Key type lgssc unregistered [ 8177.278850] LNet: 253009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8177.284842] LNetError: 253009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8178.364028] LNet: Removed LNI 192.168.202.127@tcp [ 8179.339202] Key type .llcrypt unregistered [ 8179.342535] Key type ._llcrypt unregistered [ 8223.555744] Key type ._llcrypt registered [ 8223.559313] Key type .llcrypt registered [ 8223.647339] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8225.537227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8225.613739] alg: No test for adler32 (adler32-zlib) [ 8226.748822] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 8226.951650] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8228.603872] Key type lgssc registered [ 8229.897325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8241.109954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8241.141557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8242.574651] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8242.604271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8242.899816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8249.755982] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8260.273891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8260.411262] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8260.465288] Lustre: 255219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8260.844315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8266.177534] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8277.010856] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8283.863469] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8291.256417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.563910] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8291.841275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8299.696410] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8300.334974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8307.589191] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8361.961295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8361.964045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8361.975491] LustreError: Skipped 1 previous similar message [ 8361.980962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8362.003946] Lustre: Skipped 1 previous similar message [ 8367.074811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8367.086479] Lustre: Skipped 2 previous similar messages [ 8368.099666] Lustre: server umount lustre-OST0000 complete [ 8372.458298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8386.948940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8387.401755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8392.514287] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8392.531524] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8392.534701] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8393.918278] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8401.110580] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8443.873680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8443.877959] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8443.882240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8443.893453] Lustre: Skipped 1 previous similar message [ 8446.386571] Lustre: server umount lustre-OST0000 complete [ 8450.529188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8450.537203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8450.548105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8450.550889] Lustre: Skipped 1 previous similar message [ 8454.116089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8456.630214] Lustre: server umount lustre-MDT0000 complete [ 8460.947691] LustreError: 257080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786082971 with bad export cookie 14246401255207057043 [ 8460.954142] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8460.967237] LustreError: 257080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8461.394242] Lustre: server umount lustre-MDT0001 complete [ 8477.222635] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8481.120625] Key type lgssc unregistered [ 8481.573862] LNet: 260645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8481.579214] LNetError: 260645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8481.600320] LNet: Removed LNI 192.168.202.127@tcp [ 8482.667770] Key type .llcrypt unregistered [ 8482.670230] Key type ._llcrypt unregistered [ 8510.042540] Key type ._llcrypt registered [ 8510.046462] Key type .llcrypt registered [ 8510.222476] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8512.041879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8512.065518] alg: No test for adler32 (adler32-zlib) [ 8513.132149] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 8513.622272] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8515.343202] Key type lgssc registered [ 8516.998432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8525.557309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8525.594681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.098840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8531.280918] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8537.570074] LustreError: 262093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.595525] LustreError: 262093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8541.018315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.435278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8547.020929] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8556.105595] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8561.736791] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8569.163423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8569.622248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8572.736724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8576.922606] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8586.546711] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8611.813281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8611.810956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8611.850636] Lustre: Skipped 1 previous similar message [ 8616.794390] Lustre: server umount lustre-OST0000 complete [ 8622.052410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8622.061106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8622.072977] Lustre: Skipped 1 previous similar message [ 8622.093175] Lustre: Skipped 2 previous similar messages [ 8627.169418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8627.184693] Lustre: Skipped 1 previous similar message [ 8632.291518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8635.360129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8635.752545] Lustre: server umount lustre-MDT0000 complete [ 8637.420342] LustreError: 262821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8637.442780] LustreError: 262821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8639.914823] LustreError: 263936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083150 with bad export cookie 4700989496976834000 [ 8639.919806] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8639.929080] LustreError: 263936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8640.372381] Lustre: server umount lustre-MDT0001 complete [ 8652.182869] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8654.858937] Key type lgssc unregistered [ 8655.175549] LNet: 266398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8655.197187] LNetError: 266398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8655.231713] LNet: Removed LNI 192.168.202.127@tcp [ 8656.332574] Key type .llcrypt unregistered [ 8656.336386] Key type ._llcrypt unregistered [ 8674.432533] Key type ._llcrypt registered [ 8674.440615] Key type .llcrypt registered [ 8674.588896] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8675.970869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8676.020787] alg: No test for adler32 (adler32-zlib) [ 8677.018499] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 8677.267779] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8678.936362] Key type lgssc registered [ 8680.337412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8688.962117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8688.990100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8690.568331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8696.054482] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8699.888236] LustreError: 267621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.907918] LustreError: 267621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8704.995045] LustreError: 267620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8705.482963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.961573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8711.081386] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8718.856363] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8724.573803] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8730.863974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8731.115934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8735.218163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8738.049910] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8746.709743] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8760.816025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8760.821881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.829860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8760.865749] Lustre: Skipped 1 previous similar message [ 8765.044994] Lustre: server umount lustre-OST0000 complete [ 8771.045749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.060497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.084996] Lustre: Skipped 1 previous similar message [ 8771.123137] Lustre: Skipped 2 previous similar messages [ 8776.161157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8776.174893] Lustre: Skipped 1 previous similar message [ 8781.281424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8781.293196] Lustre: Skipped 1 previous similar message [ 8783.329321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8783.639426] Lustre: server umount lustre-MDT0000 complete [ 8786.412960] LustreError: 267617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.429456] LustreError: 267617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8788.179910] LustreError: 267600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083298 with bad export cookie 8531966096644952706 [ 8788.183932] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8788.190245] LustreError: 267600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8788.631858] Lustre: server umount lustre-MDT0001 complete [ 8800.988849] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8804.607473] Key type lgssc unregistered [ 8804.849271] LNet: 271300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8804.866854] LNetError: 271300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8805.925539] LNet: Removed LNI 192.168.202.127@tcp [ 8807.502937] Key type .llcrypt unregistered [ 8807.504491] Key type ._llcrypt unregistered [ 8835.627570] Key type ._llcrypt registered [ 8835.632725] Key type .llcrypt registered [ 8835.814224] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8837.178884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8837.196866] alg: No test for adler32 (adler32-zlib) [ 8838.160972] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 8838.426650] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8840.199200] Key type lgssc registered [ 8841.759766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8850.690660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8850.710432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.104303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8856.543271] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8861.673579] LustreError: 272721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8861.691045] LustreError: 272721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8865.813730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.122214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8870.954838] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8879.290875] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8885.061854] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8892.447210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8892.807470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8896.880806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8900.742099] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8910.269300] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8936.930509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.956163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8936.965580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8940.311649] Lustre: server umount lustre-OST0000 complete [ 8947.169232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.179918] Lustre: Skipped 2 previous similar messages [ 8947.181255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8947.201845] Lustre: Skipped 2 previous similar messages [ 8950.164497] Lustre: server umount lustre-MDT0000 complete [ 8952.298572] LustreError: 272720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8952.318319] LustreError: 272720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8954.518615] LustreError: 272701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083464 with bad export cookie 14905978137861291753 [ 8954.535895] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8954.990356] Lustre: server umount lustre-MDT0001 complete [ 8967.774187] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8970.567580] Key type lgssc unregistered [ 8970.836879] LNet: 277025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8970.844394] LNetError: 277025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8971.887169] LNet: Removed LNI 192.168.202.127@tcp [ 8972.793619] Key type .llcrypt unregistered [ 8972.795627] Key type ._llcrypt unregistered [ 8990.209697] Key type ._llcrypt registered [ 8990.212731] Key type .llcrypt registered [ 8990.331714] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8991.826623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8991.881249] alg: No test for adler32 (adler32-zlib) [ 8992.983801] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 8993.172216] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8994.847210] Key type lgssc registered [ 8995.749168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9003.045586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9003.076527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9004.853949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9009.325872] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9015.268029] LustreError: 278262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9015.294654] LustreError: 278262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9016.949482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9017.272784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9021.321264] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9028.854164] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9034.398449] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9040.799988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9041.155136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9047.677752] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9050.366605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9054.404940] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9065.956286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9065.958136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9065.988603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9071.075517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9071.090659] Lustre: Skipped 2 previous similar messages [ 9072.065186] Lustre: server umount lustre-OST0000 complete [ 9076.203444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9076.221659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9076.227142] Lustre: Skipped 2 previous similar messages [ 9076.239501] Lustre: Skipped 1 previous similar message [ 9081.313939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9081.320770] Lustre: Skipped 1 previous similar message [ 9086.433728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9086.441297] Lustre: Skipped 1 previous similar message [ 9090.015189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9090.248897] Lustre: server umount lustre-MDT0000 complete [ 9091.553424] LustreError: 278275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9091.595455] LustreError: 278275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9094.106461] LustreError: 278227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083604 with bad export cookie 3338403838601771229 [ 9094.120267] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9094.125030] LustreError: 278227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9094.481326] Lustre: server umount lustre-MDT0001 complete [ 9106.765510] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 9110.808349] Key type lgssc unregistered [ 9111.298667] LNet: 281963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9111.307800] LNetError: 281963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9111.338026] LNet: Removed LNI 192.168.202.127@tcp [ 9112.614945] Key type .llcrypt unregistered [ 9112.618560] Key type ._llcrypt unregistered [ 9126.464313] Key type ._llcrypt registered [ 9126.472478] Key type .llcrypt registered [ 9126.615023] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9147.233572] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 9163.443295] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9164.964438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9165.176302] alg: No test for adler32 (adler32-zlib) [ 9166.311952] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 9166.656165] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 9168.463178] Key type lgssc registered [ 9170.116583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9209.948429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9210.003607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9211.638348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9211.678404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9211.799218] Lustre: lustre-MDT0000: new disk, initializing [ 9211.913709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9211.940537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9217.471593] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9230.583994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9230.852038] Lustre: 285523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9231.044883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9231.055046] Lustre: Skipped 1 previous similar message [ 9231.161821] Lustre: lustre-MDT0001: new disk, initializing [ 9231.324499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9231.447193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9231.474626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9237.492641] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9248.286780] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9254.414226] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9262.246551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9262.456658] Lustre: lustre-OST0000: new disk, initializing [ 9262.460946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9262.466819] Lustre: 287491:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9262.567795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9269.412700] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9272.457479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9272.470935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9272.552206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9279.891801] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9285.144708] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9285.464180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9290.641135] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9291.058939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9293.282159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9293.290658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9293.301470] Lustre: Skipped 1 previous similar message [ 9293.305831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9293.318342] Lustre: Skipped 1 previous similar message [ 9298.402213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9298.411348] Lustre: Skipped 1 previous similar message [ 9299.356957] Lustre: server umount lustre-OST0000 complete [ 9308.130747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9308.145427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9308.176489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9311.392877] Lustre: server umount lustre-MDT0000 complete [ 9315.951440] LustreError: 285516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083826 with bad export cookie 14284359441752852755 [ 9315.965106] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9315.968333] LustreError: 285516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9316.510638] Lustre: server umount lustre-MDT0001 complete [ 9327.176726] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 02:23:55 (1786083835) [ 9334.771987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9335.418477] LustreError: 289904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9335.450577] LustreError: 289904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9335.553578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9339.552976] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9340.896113] LustreError: 289905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9345.006477] LustreError: 289904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9347.771379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9348.131811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9353.042238] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9361.950738] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9367.083095] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9374.504807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9374.959720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9381.952670] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9390.239719] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9403.874143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9403.875562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9403.909267] Lustre: Skipped 1 previous similar message [ 9403.910209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9403.910222] Lustre: Skipped 1 previous similar message [ 9408.996368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9409.004928] Lustre: Skipped 1 previous similar message [ 9409.697583] Lustre: server umount lustre-OST0000 complete [ 9414.117927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9414.143152] Lustre: Skipped 1 previous similar message [ 9419.233239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9419.240842] Lustre: Skipped 4 previous similar messages [ 9419.856869] Lustre: server umount lustre-MDT0000 complete [ 9423.448720] LustreError: 289886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786083933 with bad export cookie 14284359441752853637 [ 9423.464427] LustreError: 289886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9423.467633] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9423.740891] Lustre: server umount lustre-MDT0001 complete [ 9433.903223] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 9436.809025] Key type lgssc unregistered [ 9437.077374] LNet: 293460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9437.090443] LNetError: 293460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9437.104835] LNet: Removed LNI 192.168.202.127@tcp [ 9438.126118] Key type .llcrypt unregistered [ 9438.129753] Key type ._llcrypt unregistered [ 9462.798503] Key type ._llcrypt registered [ 9462.805809] Key type .llcrypt registered [ 9462.990307] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 9479.037216] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9480.748252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9480.795549] alg: No test for adler32 (adler32-zlib) [ 9481.930943] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [ 9482.191143] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 9483.887401] Key type lgssc registered [ 9485.358052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9538.379400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9538.420957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9539.703823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9539.754205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9539.846244] Lustre: lustre-MDT0000: new disk, initializing [ 9539.946295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9539.967133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9545.550148] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9556.374388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.571316] Lustre: 297287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9556.669777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9556.678313] Lustre: Skipped 1 previous similar message [ 9556.814737] Lustre: lustre-MDT0001: new disk, initializing [ 9556.939053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9556.977091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9556.997516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9561.743426] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9572.242717] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9577.801558] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9586.241071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9586.694455] Lustre: lustre-OST2710: new disk, initializing [ 9586.703642] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9586.727836] Lustre: 299256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9586.848039] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9593.483796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9593.512140] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9593.802890] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9594.698799] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9604.541965] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9611.988771] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9612.222510] Lustre: lustre-OST03e8: new disk, initializing [ 9612.226447] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9612.231895] Lustre: 300321:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9612.377595] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9619.159851] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9621.171787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9621.216494] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9621.280752] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9627.139582] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9633.310666] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9633.527158] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9638.203583] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9638.358504] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9642.587776] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9642.882985] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9647.036626] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9647.185588] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9654.244513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9654.256866] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9654.277248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.314730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.315625] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9657.334113] Lustre: Skipped 2 previous similar messages [ 9659.520636] Lustre: server umount lustre-MDT0000 complete [ 9664.784310] LustreError: 298091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084175 with bad export cookie 1501587972880895599 [ 9664.790446] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9664.801989] LustreError: 298091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9665.569602] Lustre: server umount lustre-MDT0001 complete [ 9679.873325] Lustre: server umount lustre-OST2710 complete [ 9683.810622] Lustre: 295086:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786084178/real 1786084178] req@000000007ef82be2 x1872844808066304/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786084194 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9683.849269] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9685.270619] Lustre: server umount lustre-OST03e8 complete [ 9708.453826] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 9718.854695] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9757.335044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9757.692661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9757.736738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9757.860232] Lustre: lustre-MDT0000: new disk, initializing [ 9758.008903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9758.033891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9763.492630] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9775.631652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.809073] Lustre: 305570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9775.885503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9775.895226] Lustre: Skipped 1 previous similar message [ 9776.011516] Lustre: lustre-MDT0001: new disk, initializing [ 9776.128492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9776.218604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9776.245023] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9782.481045] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9793.094308] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9799.527373] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9807.006959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9807.205179] Lustre: lustre-OST0000: new disk, initializing [ 9807.214689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9807.227566] Lustre: 307541:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9807.395328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9814.258703] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9817.162685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9817.170838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9817.291811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9824.861787] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9831.360695] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9831.555857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9836.585846] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9836.795969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9843.170659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9843.172216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9843.189677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9843.189685] Lustre: Skipped 3 previous similar messages [ 9843.234194] Lustre: Skipped 1 previous similar message [ 9845.131984] Lustre: server umount lustre-OST0000 complete [ 9852.899808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9852.909304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9852.929864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9852.942146] Lustre: Skipped 1 previous similar message [ 9858.531150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9858.542403] Lustre: Skipped 3 previous similar messages [ 9863.135220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9863.470729] Lustre: server umount lustre-MDT0000 complete [ 9867.654099] LustreError: 305564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084378 with bad export cookie 1501587972880897293 [ 9867.678562] LustreError: 305564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9867.679693] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9868.158643] Lustre: server umount lustre-MDT0001 complete [ 9877.171057] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 02:33:05 (1786084385) [ 9878.887197] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9880.677878] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 02:33:09 (1786084389) [ 9885.281492] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9915.749764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9916.168111] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9916.200611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9916.521230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9922.243604] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9932.873832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9932.916521] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9932.932508] Lustre: 311403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9937.418141] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9959.474471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9959.724982] Lustre: Denying initial registration attempt from nid 192.168.202.127@tcp, specified as failover [ 9959.735225] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9959.749147] LustreError: 312858:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9959.755874] LustreError: 312858:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9959.766813] LustreError: 312858:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9959.875778] Lustre: server umount lustre-OST0000 complete [ 9959.881635] LustreError: 312858:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9969.120791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9969.135983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9969.153646] Lustre: Skipped 1 previous similar message [ 9969.158852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9969.164927] Lustre: Skipped 2 previous similar messages [ 9981.408644] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9981.556552] Lustre: server umount lustre-MDT0000 complete [ 9984.997139] LustreError: 311410:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.026230] LustreError: 311410:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9986.257060] LustreError: 311395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084496 with bad export cookie 1501587972880898175 [ 9986.274279] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9986.284748] LustreError: 311395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9986.923893] Lustre: server umount lustre-MDT0001 complete [ 9998.105537] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10000.855964] Key type lgssc unregistered [10001.190132] LNet: 313979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10001.205949] LNetError: 313979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10001.219795] LNet: Removed LNI 192.168.202.127@tcp [10001.985298] Key type .llcrypt unregistered [10001.988522] Key type ._llcrypt unregistered [10015.589956] Key type ._llcrypt registered [10015.591936] Key type .llcrypt registered [10015.749643] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10017.561184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10017.593705] alg: No test for adler32 (adler32-zlib) [10018.776183] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [10019.124683] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10020.895206] Key type lgssc registered [10022.388802] Lustre: Echo OBD driver; http://www.lustre.org/ [10068.558746] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10077.258243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10077.271932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10078.525577] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10078.552450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10078.695220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10083.546922] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10092.679562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10092.811503] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10092.856978] Lustre: 316744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10093.148712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10099.884129] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10108.383458] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10114.601342] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10135.028426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10135.248884] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10135.434336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10142.118371] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10149.539363] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10153.953229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10153.960113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10153.982513] Lustre: Skipped 1 previous similar message [10153.994783] Lustre: Skipped 1 previous similar message [10158.431846] Lustre: server umount lustre-OST0000 complete [10164.194769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10164.212481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10164.221874] Lustre: Skipped 1 previous similar message [10168.712838] Lustre: server umount lustre-MDT0000 complete [10172.830355] LustreError: 316737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084683 with bad export cookie 14823834931497924007 [10172.835991] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10172.849816] LustreError: 316737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10173.221861] Lustre: server umount lustre-MDT0001 complete [10183.462733] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10186.576130] Key type lgssc unregistered [10186.836884] LNet: 320320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10186.845530] LNetError: 320320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10186.879850] LNet: Removed LNI 192.168.202.127@tcp [10188.203806] Key type .llcrypt unregistered [10188.206187] Key type ._llcrypt unregistered [10215.072147] Key type ._llcrypt registered [10215.078356] Key type .llcrypt registered [10215.271802] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10216.177855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10216.279077] alg: No test for adler32 (adler32-zlib) [10217.629177] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [10218.071446] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10219.815289] Key type lgssc registered [10221.314503] Lustre: Echo OBD driver; http://www.lustre.org/ [10228.891674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10228.913144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.374981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10234.772567] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10240.488508] LustreError: 321738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10240.502973] LustreError: 321738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10243.944717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10244.412460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10249.056730] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10256.758471] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10263.526837] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10273.259694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10273.623376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10280.453034] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10288.187181] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10295.776835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10295.787816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10295.806490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10299.360750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10299.368806] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10302.141991] Lustre: server umount lustre-MDT0000 complete [10304.486322] LustreError: 324229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10304.511874] LustreError: 324229:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10306.505127] LustreError: 321720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084816 with bad export cookie 10961288999132364166 [10306.507735] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10306.518825] LustreError: 321720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10309.616061] LustreError: 324229:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.626598] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10309.641062] LustreError: 324229:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10309.666712] Lustre: Skipped 1 previous similar message [10309.677967] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10309.684943] Lustre: Skipped 3 previous similar messages [10311.648498] LustreError: 323578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10311.664454] LustreError: 323578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10313.125225] Lustre: server umount lustre-MDT0001 complete [10320.373261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10329.628466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.719325] LustreError: 325321:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10330.729488] LustreError: 325321:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000229085a6 x1872845579788800/t0(0) o250->MGC192.168.202.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786084841 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10330.748227] LustreError: 325321:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10331.410115] LustreError: 323569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10331.461380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10331.532245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10336.026131] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10340.712195] LustreError: 325335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10340.734395] LustreError: 325335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10340.748608] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10344.610874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.919583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10349.227626] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10350.080681] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10356.649479] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10358.428976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10364.091619] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10365.839944] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10372.582919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10372.588646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10372.597062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10372.601155] Lustre: Skipped 2 previous similar messages [10375.655756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10380.783747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10380.792127] Lustre: Skipped 3 previous similar messages [10385.375133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10385.569671] Lustre: server umount lustre-OST0000 complete [10391.008063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.024753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10391.026078] Lustre: Skipped 2 previous similar messages [10391.031774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10391.039820] Lustre: Skipped 1 previous similar message [10403.295237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10403.862569] Lustre: server umount lustre-MDT0000 complete [10406.376785] LustreError: 325334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.417839] LustreError: 325334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10408.706175] LustreError: 321720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786084919 with bad export cookie 10961288999132371831 [10408.718309] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10408.733877] LustreError: 321720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10409.130753] Lustre: server umount lustre-MDT0001 complete [10421.279247] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10423.811132] Key type lgssc unregistered [10424.142775] LNet: 328009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10424.151144] LNetError: 328009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10424.176870] LNet: Removed LNI 192.168.202.127@tcp [10425.061123] Key type .llcrypt unregistered [10425.064077] Key type ._llcrypt unregistered [10449.367292] Key type ._llcrypt registered [10449.369311] Key type .llcrypt registered [10449.465275] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10450.576683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10450.694339] alg: No test for adler32 (adler32-zlib) [10451.677524] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [10451.859904] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10453.543196] Key type lgssc registered [10454.702551] Lustre: Echo OBD driver; http://www.lustre.org/ [10462.571576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10462.588052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.137268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10468.675646] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10474.470168] LustreError: 329425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10474.509502] LustreError: 329425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10479.401434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.586071] LustreError: 329426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10479.838149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10484.009332] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10491.358154] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10496.631652] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10500.580146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10500.587696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10500.593257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10500.606784] Lustre: Skipped 1 previous similar message [10500.634897] Lustre: Skipped 1 previous similar message [10504.736814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10504.736814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10506.224429] Lustre: server umount lustre-MDT0000 complete [10509.793752] LustreError: 329426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10509.830355] LustreError: 329426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10509.873652] LustreError: 329407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085020 with bad export cookie 15869666247156975720 [10509.882629] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10509.889649] LustreError: 329407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10510.348784] Lustre: server umount lustre-MDT0001 complete [10525.519499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10525.772223] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10525.820730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10526.062498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10530.409170] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10539.087733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10539.225782] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10539.255644] Lustre: 332010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10539.549373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10544.218405] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10551.321262] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10556.521565] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10565.942240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10566.200076] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10566.222094] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10566.233815] LustreError: 333896:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10566.259089] LustreError: 333896:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10566.272770] LustreError: 333896:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10566.392241] Lustre: server umount lustre-OST0000 complete [10566.394750] LustreError: 333896:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10577.480754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10577.852612] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10578.168207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10584.680337] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10585.359886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10591.826572] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10599.166519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10599.275319] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10599.360615] Lustre: lustre-OST0001: new disk, initializing [10599.363922] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10599.368616] Lustre: 335439:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10599.475126] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10606.208557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10606.232061] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10606.372289] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10606.497952] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10614.273775] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10621.928882] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10621.962927] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10621.972069] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10621.986471] Lustre: Skipped 1 previous similar message [10624.292902] Lustre: server umount lustre-OST0001 complete [10627.058310] LustreError: 334362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10627.084417] LustreError: 334362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10631.650666] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10631.651904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10631.665064] LustreError: Skipped 2 previous similar messages [10631.673907] Lustre: Skipped 1 previous similar message [10631.679450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10632.162431] LustreError: 334364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10632.185264] LustreError: 334364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10634.961625] Lustre: server umount lustre-OST0000 complete [10641.896807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10641.900206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10641.934531] Lustre: Skipped 1 previous similar message [10641.940214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10641.942792] Lustre: Skipped 3 previous similar messages [10645.662152] Lustre: server umount lustre-MDT0000 complete [10647.523924] LustreError: 332024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10647.551340] LustreError: 332024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10649.809432] LustreError: 332003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085160 with bad export cookie 15869666247156976273 [10649.817630] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10649.818789] LustreError: 332003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10650.223510] Lustre: server umount lustre-MDT0001 complete [10663.217268] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10666.500545] Key type lgssc unregistered [10666.851909] LNet: 337398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10666.872347] LNetError: 337398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10666.893924] LNet: Removed LNI 192.168.202.127@tcp [10667.928755] Key type .llcrypt unregistered [10667.930925] Key type ._llcrypt unregistered [10737.932459] Key type ._llcrypt registered [10737.936979] Key type .llcrypt registered [10738.052830] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [10753.822142] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10755.120508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10755.143325] alg: No test for adler32 (adler32-zlib) [10756.363904] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [10756.641129] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10758.335216] Key type lgssc registered [10759.693907] Lustre: Echo OBD driver; http://www.lustre.org/ [10805.387548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10805.413657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10806.774043] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10806.847908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10806.955636] Lustre: lustre-MDT0000: new disk, initializing [10807.103587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10807.121918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10811.832886] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10824.312079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10824.459657] Lustre: 342337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10824.503781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10824.507262] Lustre: Skipped 1 previous similar message [10824.581625] Lustre: lustre-MDT0001: new disk, initializing [10824.692857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10824.742768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10824.759308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10830.588230] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10841.242509] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10847.589300] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10854.311911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10854.657671] Lustre: lustre-OST0000: new disk, initializing [10854.663477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10854.675267] Lustre: 344302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10854.794939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10861.113556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10861.122690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10861.231116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10862.375822] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10872.359391] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10877.657263] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10877.940717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10882.798406] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10883.066597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10887.143055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10887.152463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10887.171373] Lustre: Skipped 1 previous similar message [10887.179501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10891.114572] Lustre: server umount lustre-OST0000 complete [10896.356852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10896.361182] LustreError: Skipped 1 previous similar message [10896.363885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.382067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10896.384907] Lustre: Skipped 1 previous similar message [10897.376553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10897.395737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10897.404742] Lustre: Skipped 1 previous similar message [10901.332061] Lustre: server umount lustre-MDT0000 complete [10905.087432] LustreError: 342326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085415 with bad export cookie 11435667745105959452 [10905.097067] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10905.107660] LustreError: 342326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10905.496459] Lustre: server umount lustre-MDT0001 complete [10914.041250] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:50:22 (1786085422) [10921.369995] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:50:30 (1786085430) [10930.327870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10930.659303] LustreError: 347058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.678399] LustreError: 347058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10930.737588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10935.791594] LustreError: 347059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.899086] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10940.897421] LustreError: 347058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.471903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10945.824680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10951.066476] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10959.244413] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10965.667666] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10972.052551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10972.379912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10978.902959] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10985.707026] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10997.241215] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10999.020109] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11000.634242] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11002.627983] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11004.803479] Lustre: Failing over lustre-MDT0000 [11005.142310] Lustre: server umount lustre-MDT0000 complete [11006.455526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11006.459965] LustreError: 347058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.484164] Lustre: Skipped 1 previous similar message [11006.539785] LustreError: 347058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11011.552984] LustreError: 347059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11011.578093] LustreError: 347059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11021.792437] Lustre: 340559:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786085516/real 1786085516] req@00000000ea5956b6 x1872846145000704/t0(0) o400->MGC192.168.202.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786085532 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11021.806828] LustreError: 347059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11021.820947] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11021.857111] LustreError: 347059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11025.575927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11032.032691] LustreError: 340555:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ea5956b6 x1872846145005568/t0(0) o250->MGC192.168.202.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11032.492285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11032.562149] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11033.226486] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11037.231922] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11037.677871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11037.707026] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11037.773822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11048.351745] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11050.051467] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11054.142730] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11056.015824] Lustre: Failing over lustre-MDT0000 [11056.457110] Lustre: server umount lustre-MDT0000 complete [11058.146135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11058.164449] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11058.167383] LustreError: 348045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11058.167394] LustreError: 348045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [11058.253892] Lustre: Skipped 3 previous similar messages [11074.530127] Lustre: 340557:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786085568/real 1786085568] req@00000000e2ffeb8f x1872846145021312/t0(0) o400->MGC192.168.202.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786085584 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11074.568736] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11076.280854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11084.771411] LustreError: 340555:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000064d1f157 x1872846145026688/t0(0) o250->MGC192.168.202.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11085.150903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11085.197043] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11085.436572] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11090.091659] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11090.424468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11090.436217] Lustre: Skipped 2 previous similar messages [11090.475315] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11090.515034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11099.160240] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11100.916867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11110.883951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11110.888032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11110.902907] Lustre: Skipped 1 previous similar message [11110.930541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11112.996058] Lustre: server umount lustre-MDT0000 complete [11116.006995] LustreError: 348045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11116.034996] LustreError: 348045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [11116.670904] LustreError: 347038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085627 with bad export cookie 11435667745105962161 [11116.680764] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11116.697819] LustreError: 347038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11117.311847] Lustre: server umount lustre-MDT0001 complete [11130.874272] Lustre: server umount lustre-OST0000 complete [11150.959859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11151.511927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11157.331481] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11166.231906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11171.141094] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11180.217414] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11187.226873] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11194.113507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11196.597856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11201.222717] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11209.132977] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11221.475162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11221.475807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11221.484443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11221.484451] Lustre: Skipped 2 previous similar messages [11221.552653] Lustre: Skipped 2 previous similar messages [11227.542947] Lustre: server umount lustre-OST0000 complete [11233.248296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11233.266922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11233.277688] Lustre: Skipped 3 previous similar messages [11238.590529] Lustre: server umount lustre-MDT0000 complete [11241.957868] LustreError: 355909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.980615] LustreError: 355909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11242.667608] LustreError: 353395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085753 with bad export cookie 11435667745105962910 [11242.677824] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11242.680686] LustreError: 353395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11243.371449] Lustre: server umount lustre-MDT0001 complete [11256.989509] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11260.538926] Key type lgssc unregistered [11261.023423] LNet: 356950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11261.048391] LNetError: 356950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11261.089623] LNet: Removed LNI 192.168.202.127@tcp [11262.327137] Key type .llcrypt unregistered [11262.329480] Key type ._llcrypt unregistered [11292.742818] Key type ._llcrypt registered [11292.749949] Key type .llcrypt registered [11292.907122] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11294.127668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11294.141515] alg: No test for adler32 (adler32-zlib) [11295.561735] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [11296.053065] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11297.743315] Key type lgssc registered [11299.640286] Lustre: Echo OBD driver; http://www.lustre.org/ [11309.047834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11309.090882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.761915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11315.744152] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11321.318934] LustreError: 358469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.340585] LustreError: 358469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11326.133404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11326.432635] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11326.601061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11332.530668] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11340.946031] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11347.490577] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11355.603781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11356.138369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11363.318210] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11371.760086] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11380.786184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11396.066475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11396.075363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11396.090562] Lustre: Skipped 2 previous similar messages [11396.116179] Lustre: Skipped 2 previous similar messages [11401.193889] LustreError: 358469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11401.209516] LustreError: 358469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11401.362968] Lustre: server umount lustre-MDT0000 complete [11405.474304] LustreError: 358449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786085915 with bad export cookie 14726838763349054626 [11405.493642] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11405.500229] LustreError: 358449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11405.977458] Lustre: server umount lustre-MDT0001 complete [11420.712972] Lustre: server umount lustre-OST0000 complete [11437.172294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11437.789073] LustreError: 362202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11437.877308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11443.171981] LustreError: 362203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11443.219784] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11448.292482] LustreError: 362202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11453.220089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.788784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11459.264666] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11469.531942] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11476.657925] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11485.423794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11485.791568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11494.483897] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11505.206808] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11508.351669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11524.070843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11524.072052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11524.094806] Lustre: Skipped 2 previous similar messages [11524.112440] Lustre: Skipped 2 previous similar messages [11529.190870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11529.221677] Lustre: Skipped 2 previous similar messages [11530.171557] Lustre: server umount lustre-MDT0000 complete [11534.313461] LustreError: 362203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11534.343651] LustreError: 362203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11534.447054] LustreError: 362184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786086044 with bad export cookie 14726838763349055739 [11534.456276] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11534.462697] LustreError: 362184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11535.022862] Lustre: server umount lustre-MDT0001 complete [11550.567787] Lustre: server umount lustre-OST0000 complete [11568.666981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11569.299499] LustreError: 366129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11569.448796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11575.095327] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11584.953161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11591.102887] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11599.122737] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11605.963326] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11613.699729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11614.163778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11614.169558] Lustre: Skipped 1 previous similar message [11615.239364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11622.192190] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11630.857476] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11640.808516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11640.812137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11640.833848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11643.892133] Lustre: server umount lustre-OST0000 complete [11651.040257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11651.045251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11651.062183] Lustre: Skipped 2 previous similar messages [11651.074762] Lustre: Skipped 2 previous similar messages [11656.049932] Lustre: server umount lustre-MDT0000 complete [11656.160365] LustreError: 367421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11656.194285] LustreError: 367421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11661.296322] LustreError: 366110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786086171 with bad export cookie 14726838763349056761 [11661.300187] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11661.313436] LustreError: 366110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11661.700297] Lustre: server umount lustre-MDT0001 complete [11676.835255] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11679.875491] Key type lgssc unregistered [11680.362512] LNet: 369664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11680.370714] LNetError: 369664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11680.411880] LNet: Removed LNI 192.168.202.127@tcp [11682.130617] Key type .llcrypt unregistered [11682.140360] Key type ._llcrypt unregistered [11712.522180] Key type ._llcrypt registered [11712.524423] Key type .llcrypt registered [11712.615482] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11714.044496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11714.062358] alg: No test for adler32 (adler32-zlib) [11715.165403] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [11715.405297] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11717.096899] Key type lgssc registered [11718.651543] Lustre: Echo OBD driver; http://www.lustre.org/ [11728.231777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11728.254759] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.261444] LustreError: 371107:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11728.285202] LustreError: 371107:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11728.295307] LustreError: 371107:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11728.304694] LustreError: 371107:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11728.316317] LustreError: 371107:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11737.268218] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11737.272686] LustreError: 371505:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11737.286757] LustreError: 371505:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11737.291364] LustreError: 371505:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11737.295957] LustreError: 371505:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11737.301265] LustreError: 371505:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11759.405898] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11762.959783] Key type lgssc unregistered [11763.294623] LNet: 372509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11763.308169] LNetError: 372509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11763.328168] LNet: Removed LNI 192.168.202.127@tcp [11764.186819] Key type .llcrypt unregistered [11764.188953] Key type ._llcrypt unregistered [11784.206579] Key type ._llcrypt registered [11784.213929] Key type .llcrypt registered [11784.316549] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [11801.084687] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11802.244334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11802.322062] alg: No test for adler32 (adler32-zlib) [11803.413376] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [11803.736099] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11805.407326] Key type lgssc registered [11806.479405] Lustre: Echo OBD driver; http://www.lustre.org/ [11849.780615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11849.856864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11851.180900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11851.224865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11851.372146] Lustre: lustre-MDT0000: new disk, initializing [11851.536876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11851.603912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11857.116985] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11869.890277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11870.018389] Lustre: 375672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11870.083587] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11870.087203] Lustre: Skipped 1 previous similar message [11870.169869] Lustre: lustre-MDT0001: new disk, initializing [11870.302466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11870.439304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11870.465994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11875.821383] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11886.130931] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11892.990115] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11899.986762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11900.378857] Lustre: lustre-OST0000: new disk, initializing [11900.382732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11900.397104] Lustre: 377642:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11900.536444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11908.029853] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11910.851589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11910.874409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11911.066856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11920.454660] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11925.465977] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11925.735599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11930.982548] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11931.162874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11936.738446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11936.743394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11936.752446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11936.773962] Lustre: Skipped 1 previous similar message [11936.798713] Lustre: Skipped 1 previous similar message [11939.249560] Lustre: server umount lustre-OST0000 complete [11946.984675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11946.984919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11947.004249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11947.016907] Lustre: Skipped 1 previous similar message [11947.027242] Lustre: Skipped 1 previous similar message [11949.903938] Lustre: server umount lustre-MDT0000 complete [11953.839095] LustreError: 375664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786086464 with bad export cookie 13452720484789411186 [11953.842304] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11953.847154] LustreError: 375664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11954.323921] Lustre: server umount lustre-MDT0001 complete [11962.761923] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11964.352654] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:07:53 (1786086473) [11971.133307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11971.812209] LustreError: 380103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11971.840711] LustreError: 380103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11971.905638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11976.619270] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11977.186045] LustreError: 380104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11982.307834] LustreError: 380103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11986.398244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11986.778850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11991.373541] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11998.924320] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12005.095634] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12012.344214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12012.526667] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [12012.699715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12020.390901] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12031.226519] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12041.006364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12041.427704] Lustre: lustre-OST0001: new disk, initializing [12041.435862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12041.456946] Lustre: 382941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12041.626026] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12049.053626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12049.058910] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12049.184428] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12051.002411] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12062.953300] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12069.861182] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12069.881988] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12069.890709] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12069.903145] Lustre: Skipped 1 previous similar message [12074.995743] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12075.004887] Lustre: Skipped 3 previous similar messages [12075.421414] Lustre: server umount lustre-OST0001 complete [12076.156377] LustreError: 381927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12085.228017] LustreError: 381925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12085.240589] LustreError: 381925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12095.457027] LustreError: 381928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12095.460815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12095.462357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12095.480603] LustreError: 381928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12100.926625] Lustre: server umount lustre-OST0000 complete [12109.793099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12109.801822] LustreError: Skipped 1 previous similar message [12109.808459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12109.818646] Lustre: Skipped 1 previous similar message [12112.897459] Lustre: server umount lustre-MDT0000 complete [12115.937928] LustreError: 380098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12115.966748] LustreError: 380098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12117.213322] LustreError: 383749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786086627 with bad export cookie 13452720484789412068 [12117.227843] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12117.235221] LustreError: 383749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12123.715705] Lustre: server umount lustre-MDT0001 complete [12137.452547] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12140.783247] Key type lgssc unregistered [12141.065661] LNet: 384932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12141.075677] LNetError: 384932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12141.098137] LNet: Removed LNI 192.168.202.127@tcp [12142.216154] Key type .llcrypt unregistered [12142.218065] Key type ._llcrypt unregistered [12200.061134] Key type ._llcrypt registered [12200.062777] Key type .llcrypt registered [12200.187988] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12201.636106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12202.079446] alg: No test for adler32 (adler32-zlib) [12203.537774] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [12203.963134] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [12205.727405] Key type lgssc registered [12207.388638] Lustre: Echo OBD driver; http://www.lustre.org/ [12217.769042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12217.809979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12219.273768] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12219.295649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12219.439881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12224.185129] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12234.561995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12234.657705] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12234.681995] Lustre: 387732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12234.785231] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12234.974063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12239.930659] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12248.487576] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12254.750512] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12260.843769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12260.849977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12260.856075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12260.872993] Lustre: Skipped 1 previous similar message [12260.903654] Lustre: Skipped 1 previous similar message [12262.943126] Lustre: server umount lustre-MDT0000 complete [12267.121934] LustreError: 387727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786086777 with bad export cookie 9380897243863772633 [12267.128807] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12267.133554] LustreError: 387727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12267.550976] Lustre: server umount lustre-MDT0001 complete [12280.276733] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:13:08 (1786086788) [12287.467749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12287.979683] LustreError: 390246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12287.993174] LustreError: 390246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12288.025398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12292.182492] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12293.102967] LustreError: 390246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12293.131365] LustreError: 390246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12298.227070] LustreError: 390247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12302.416462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12302.804291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12307.611981] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12314.318253] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12319.418168] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12325.558250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12325.713137] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12325.928241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12333.697249] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12344.054642] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12353.854361] Lustre: Permanently deactivating lustre-OST0000 [12353.872086] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12359.728067] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12361.585856] LustreError: 393186:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12361.598765] LustreError: 393186:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12361.605730] LustreError: 393186:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12364.686986] LustreError: 393236:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12364.690772] LustreError: 393236:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12372.578945] Lustre: server umount lustre-OST0000 complete [12375.272427] LustreError: 393485:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12375.276919] LustreError: 393485:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12378.087053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12378.090849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12378.095064] Lustre: Skipped 1 previous similar message [12378.109832] Lustre: Skipped 1 previous similar message [12383.203061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12383.218668] Lustre: Skipped 1 previous similar message [12388.325227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12388.328228] Lustre: Skipped 1 previous similar message [12391.391567] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12391.638716] Lustre: server umount lustre-MDT0000 complete [12393.448791] LustreError: 390246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12396.823302] LustreError: 390226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786086907 with bad export cookie 9380897243863773214 [12396.834190] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12396.857741] LustreError: 390226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12407.639232] Lustre: server umount lustre-MDT0001 complete [12417.559083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12422.781383] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12425.652853] LustreError: 394646:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12432.686201] Lustre: 394985:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.127@tcp [12434.773046] Lustre: server umount MGS complete [12446.118792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12446.534425] LustreError: 395709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12446.554805] LustreError: 395709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12446.562666] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12446.569232] Lustre: Skipped 1 previous similar message [12446.635437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12451.104891] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12456.928890] LustreError: 395710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.947478] LustreError: 395710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12459.769874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12460.099124] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12464.859700] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12472.897545] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12478.739429] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12480.701908] Lustre: Permanently reactivating lustre-OST0000 [12480.718313] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12480.721096] Lustre: Skipped 2 previous similar messages [12511.200110] LustreError: 397541:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12518.345535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12518.743531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12518.750792] Lustre: Skipped 1 previous similar message [12525.979442] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12534.710535] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12541.410951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12541.412417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12541.442585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12541.455260] Lustre: Skipped 1 previous similar message [12542.070118] Lustre: server umount lustre-OST0000 complete [12546.528437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12546.538690] Lustre: Skipped 2 previous similar messages [12552.659474] Lustre: server umount lustre-MDT0000 complete [12556.377540] LustreError: 395691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087066 with bad export cookie 9380897243863774264 [12556.379300] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12556.387285] LustreError: 395691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12556.657339] Lustre: server umount lustre-MDT0001 complete [12568.736288] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12571.679783] Key type lgssc unregistered [12571.965991] LNet: 400123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12571.970986] LNetError: 400123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12573.029973] LNet: Removed LNI 192.168.202.127@tcp [12573.993967] Key type .llcrypt unregistered [12573.996134] Key type ._llcrypt unregistered [12592.861636] Key type ._llcrypt registered [12592.863499] Key type .llcrypt registered [12593.028559] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [12608.038682] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12608.997829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12609.112673] alg: No test for adler32 (adler32-zlib) [12610.342386] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [12610.574897] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [12612.240659] Key type lgssc registered [12613.343743] Lustre: Echo OBD driver; http://www.lustre.org/ [12653.068277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12653.090087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12654.385203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12654.426855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12654.575432] Lustre: lustre-MDT0000: new disk, initializing [12654.690945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12654.733921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12659.993749] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12669.766569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12669.942983] Lustre: 403289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12669.998728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12670.002197] Lustre: Skipped 1 previous similar message [12670.087932] Lustre: lustre-MDT0001: new disk, initializing [12670.164116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12670.246463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12670.255873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12676.132811] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12689.322961] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12695.621241] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12705.074709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12705.569961] Lustre: lustre-OST0000: new disk, initializing [12705.573757] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12705.584628] Lustre: 405258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12705.725877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12712.948066] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12713.090155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12713.098896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12713.250832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12722.862992] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12728.356484] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12728.625803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12733.934374] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12734.220951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12739.043655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12739.044775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12739.063495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12742.607422] Lustre: server umount lustre-OST0000 complete [12747.232691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12747.248329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12747.257563] Lustre: Skipped 1 previous similar message [12747.266327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12747.274141] Lustre: Skipped 1 previous similar message [12749.281449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12749.282195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12752.963926] Lustre: server umount lustre-MDT0000 complete [12757.439374] LustreError: 403280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087267 with bad export cookie 6884368914157442889 [12757.454636] LustreError: 403280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12757.455156] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12757.867415] Lustre: server umount lustre-MDT0001 complete [12768.072097] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12770.543410] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:21:18 (1786087278) [12777.909232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12778.524498] LustreError: 407722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12778.547453] LustreError: 407722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12778.638245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12783.023603] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12784.097151] LustreError: 407723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12788.197206] LustreError: 407722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12791.938478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12792.269794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12796.924643] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12804.798416] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12810.815375] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12818.379743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12818.888927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12824.995970] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12834.193934] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12836.284320] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12838.897223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12844.512294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12844.519282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12844.537465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12844.541715] Lustre: Skipped 1 previous similar message [12847.584616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12849.219545] Lustre: server umount lustre-OST0000 complete [12853.728516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12853.732744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12853.749445] Lustre: Skipped 1 previous similar message [12853.754139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12853.761643] Lustre: Skipped 2 previous similar messages [12859.340362] Lustre: server umount lustre-MDT0000 complete [12862.947426] LustreError: 408450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12862.961307] LustreError: 408450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12863.054673] LustreError: 407703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087373 with bad export cookie 6884368914157443771 [12863.062569] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12863.527918] Lustre: server umount lustre-MDT0001 complete [12874.466213] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12877.494643] Key type lgssc unregistered [12877.757400] LNet: 411446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12877.765359] LNetError: 411446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12877.789974] LNet: Removed LNI 192.168.202.127@tcp [12878.749779] Key type .llcrypt unregistered [12878.752643] Key type ._llcrypt unregistered [12904.422776] Key type ._llcrypt registered [12904.429108] Key type .llcrypt registered [12904.568413] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12904.974329] Key type .llcrypt unregistered [12904.977572] Key type ._llcrypt unregistered [12923.387668] Key type ._llcrypt registered [12923.389966] Key type .llcrypt registered [12923.501066] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12925.010659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12925.056122] alg: No test for adler32 (adler32-zlib) [12926.186576] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [12926.427443] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [12928.111618] Key type lgssc registered [12929.005771] Lustre: Echo OBD driver; http://www.lustre.org/ [12935.976716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12935.995979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12937.335310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12941.455492] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12946.922298] LustreError: 413660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12946.943479] LustreError: 413660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12949.595815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12949.983697] LustreError: 413659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12950.025269] LustreError: 413659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12950.048356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12955.113029] LustreError: 413660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12955.141786] LustreError: 413660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12956.111933] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12960.231132] LustreError: 413656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12960.256070] LustreError: 413656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12962.288024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12963.773542] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12967.399697] LustreError: 413659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12967.426400] LustreError: 413659:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12971.130424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12971.477987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12976.183071] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12986.858406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12986.885456] Lustre: Skipped 1 previous similar message [12986.886304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12986.910322] Lustre: Skipped 1 previous similar message [12991.894744] Lustre: server umount lustre-OST0000 complete [12997.090439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12997.093827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12997.107486] Lustre: Skipped 1 previous similar message [12997.136325] Lustre: Skipped 1 previous similar message [12999.039509] Lustre: server umount lustre-MDT0000 complete [13002.212661] LustreError: 413659:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13003.456133] LustreError: 415402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087513 with bad export cookie 17871726024706610708 [13003.466055] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13003.471718] LustreError: 415402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13004.467469] Lustre: server umount lustre-MDT0001 complete [13016.716930] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13019.433420] Key type lgssc unregistered [13019.870308] LNet: 416806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13019.880994] LNetError: 416806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13019.914303] LNet: Removed LNI 192.168.202.127@tcp [13020.854446] Key type .llcrypt unregistered [13020.857467] Key type ._llcrypt unregistered [13047.183213] Key type ._llcrypt registered [13047.186606] Key type .llcrypt registered [13047.338893] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13049.218309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13049.283161] alg: No test for adler32 (adler32-zlib) [13050.340919] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [13050.606077] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13052.287221] Key type lgssc registered [13053.527501] Lustre: Echo OBD driver; http://www.lustre.org/ [13061.584581] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13061.601766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13077.280199] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13082.399456] LustreError: 418197:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [13109.151240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13118.446407] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13127.073661] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13134.080966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13134.771888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13134.911030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13139.941271] LustreError: 418223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13139.954556] LustreError: 418223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13140.085237] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13144.036907] LustreError: 419235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13144.049615] LustreError: 419235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13148.793528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13149.235468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13153.498276] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13161.589312] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13168.422679] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13179.878449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13179.885386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13179.893066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13179.907534] Lustre: Skipped 1 previous similar message [13179.921172] Lustre: Skipped 1 previous similar message [13185.000084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13185.006409] Lustre: Skipped 1 previous similar message [13190.111637] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13190.132967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13190.147817] Lustre: Skipped 1 previous similar message [13190.310722] Lustre: server umount lustre-OST0000 complete [13200.354978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13200.360055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13200.373172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13200.376793] Lustre: Skipped 1 previous similar message [13201.959274] Lustre: server umount lustre-MDT0000 complete [13205.476376] LustreError: 419239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13205.510937] LustreError: 419239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13205.577430] LustreError: 418203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087716 with bad export cookie 16174755791037252547 [13205.588818] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13205.595648] LustreError: 418203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13205.920743] Lustre: server umount lustre-MDT0001 complete [13216.376911] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13218.733597] Key type lgssc unregistered [13218.989817] LNet: 421768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13218.993527] LNetError: 421768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13219.015113] LNet: Removed LNI 192.168.202.127@tcp [13219.810460] Key type .llcrypt unregistered [13219.812585] Key type ._llcrypt unregistered [13247.398490] Key type ._llcrypt registered [13247.400413] Key type .llcrypt registered [13247.482653] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13248.779758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13248.845490] alg: No test for adler32 (adler32-zlib) [13249.989451] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [13250.183683] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13251.823233] Key type lgssc registered [13253.374707] Lustre: Echo OBD driver; http://www.lustre.org/ [13261.151807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13261.167708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13262.588769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13267.364771] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13272.042219] LustreError: 423189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13272.049715] LustreError: 423189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13276.781625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13277.231785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13281.319778] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13288.748174] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13294.125983] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13302.272908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13302.948615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13307.076345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13310.915542] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13320.078704] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13326.979471] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [13327.847617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13327.851816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13327.861318] Lustre: Skipped 2 previous similar messages [13327.877037] Lustre: Skipped 2 previous similar messages [13330.028589] Lustre: server umount lustre-MDT0000 complete [13332.078555] LustreError: 423185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13332.982418] LustreError: 423170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.27@tcp arrived at 1786087843 with bad export cookie 14554194043488254783 [13333.004594] LustreError: 423170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13335.401047] LustreError: 423171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786087845 with bad export cookie 14554194043488253887 [13335.405577] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13335.417816] LustreError: 423171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13338.080887] LustreError: 425885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13338.087435] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13338.103658] LustreError: 425885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13338.123122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13338.123954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13340.129072] LustreError: 423190:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13340.130901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13340.139121] LustreError: 423190:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13341.815624] Lustre: server umount lustre-OST0000 complete [13345.249567] LustreError: 423189:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13345.281385] LustreError: 423189:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13349.922882] Lustre: server umount lustre-MDT0001 complete [13361.112415] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13363.996692] Key type lgssc unregistered [13364.333622] LNet: 426875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13364.351738] LNetError: 426875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13364.376806] LNet: Removed LNI 192.168.202.127@tcp [13365.076806] Key type .llcrypt unregistered [13365.078710] Key type ._llcrypt unregistered [13391.843106] Key type ._llcrypt registered [13391.844257] Key type .llcrypt registered [13391.922165] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13393.087223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13393.130625] alg: No test for adler32 (adler32-zlib) [13394.166940] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [13394.459996] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13396.188456] Key type lgssc registered [13397.726881] Lustre: Echo OBD driver; http://www.lustre.org/ [13407.645492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13407.705657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13409.224681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13415.029515] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13419.492183] LustreError: 428296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13419.513806] LustreError: 428296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13424.622277] LustreError: 428297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13424.993829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13425.247739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13429.962078] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13438.947693] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13445.187131] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13453.726294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13454.198248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13455.228523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13461.912189] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13471.372830] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13476.840786] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13476.843806] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13476.879168] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13479.548349] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [13480.945755] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13480.962891] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13482.945719] Lustre: server umount lustre-MDT0001 complete [13484.656439] LustreError: 428290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13491.168822] LustreError: 428297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13491.169047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13491.190838] LustreError: 428297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13491.204649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13491.215321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13491.248102] Lustre: Skipped 1 previous similar message [13496.290256] LustreError: 428296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13496.291391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13496.798523] Lustre: server umount lustre-OST0000 complete [13501.729660] Lustre: server umount lustre-MDT0000 complete [13517.185116] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13519.902307] Key type lgssc unregistered [13520.164421] LNet: 431981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13520.172430] LNetError: 431981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13520.188950] LNet: Removed LNI 192.168.202.127@tcp [13521.025645] Key type .llcrypt unregistered [13521.029204] Key type ._llcrypt unregistered [13548.067512] Key type ._llcrypt registered [13548.070357] Key type .llcrypt registered [13548.195482] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [13565.268963] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13566.549525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13566.574545] alg: No test for adler32 (adler32-zlib) [13567.748697] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [13568.045536] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13569.783251] Key type lgssc registered [13571.109852] Lustre: Echo OBD driver; http://www.lustre.org/ [13614.509241] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13630.327700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13630.376071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13631.764798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13631.792108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13631.901212] Lustre: lustre-MDT0000: new disk, initializing [13632.096763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13632.120290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13637.212635] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13656.211592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13656.727535] Lustre: lustre-OST0000: new disk, initializing [13656.735389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13656.741735] Lustre: Skipped 1 previous similar message [13656.746675] Lustre: 437017:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13656.844451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13664.777985] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13664.864693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13664.878596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13664.915666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13675.055350] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13694.050289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13694.176681] Lustre: 436000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13694.252381] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13694.377819] Lustre: lustre-MDT0001: new disk, initializing [13694.530936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13694.568729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13694.577956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13699.173553] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13708.444456] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13708.692225] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13713.301251] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13713.620492] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13719.520500] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13719.532242] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13719.537027] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13719.555547] Lustre: Skipped 1 previous similar message [13719.570152] Lustre: Skipped 1 previous similar message [13723.791215] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [13725.884202] Lustre: server umount lustre-MDT0001 complete [13734.890399] LustreError: 436011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13734.920372] LustreError: 436011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13740.007937] LustreError: 436005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13740.024435] LustreError: 436005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13745.121300] LustreError: 436011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13745.128762] LustreError: 436011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13746.657121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13746.666922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13746.695672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13746.711034] Lustre: Skipped 2 previous similar messages [13750.240437] LustreError: 436010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13750.242665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13751.637582] Lustre: server umount lustre-OST0000 complete [13755.376943] LustreError: 436011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13762.000585] Lustre: server umount lustre-MDT0000 complete [13775.166692] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13778.019398] Key type lgssc unregistered [13778.307582] LNet: 440768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13778.320920] LNetError: 440768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13778.341555] LNet: Removed LNI 192.168.202.127@tcp [13779.151246] Key type .llcrypt unregistered [13779.153467] Key type ._llcrypt unregistered [13820.323762] Key type ._llcrypt registered [13820.326535] Key type .llcrypt registered [13820.455525] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [13820.740304] Key type .llcrypt unregistered [13820.743516] Key type ._llcrypt unregistered [13837.047390] Key type ._llcrypt registered [13837.049064] Key type .llcrypt registered [13837.202748] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [13838.745194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13838.855616] alg: No test for adler32 (adler32-zlib) [13839.902408] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [13840.331594] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [13841.999246] Key type lgssc registered [13842.926995] Lustre: Echo OBD driver; http://www.lustre.org/ [13870.179186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13870.199471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13871.400671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13871.422659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13871.492350] Lustre: lustre-MDT0000: new disk, initializing [13871.555110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13871.569877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13875.735992] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13885.383748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13885.485783] Lustre: 444147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13885.538530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13885.541232] Lustre: Skipped 1 previous similar message [13885.605484] Lustre: lustre-MDT0001: new disk, initializing [13885.680769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13885.712592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13885.728490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13891.036950] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13902.085687] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13908.565598] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13915.442516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13915.802950] Lustre: lustre-OST0000: new disk, initializing [13915.808696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13915.814698] Lustre: 446115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13915.921599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13922.381318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13922.385913] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13922.517525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13922.547580] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13931.226075] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13938.145836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13938.159210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13938.163644] Lustre: Skipped 2 previous similar messages [13938.177958] Lustre: Skipped 2 previous similar messages [13943.267710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13943.281124] Lustre: Skipped 2 previous similar messages [13944.093801] Lustre: server umount lustre-MDT0000 complete [13947.600329] LustreError: 444138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786088458 with bad export cookie 4125054480377659899 [13947.608600] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13947.831574] Lustre: server umount lustre-MDT0001 complete [13961.805314] Lustre: server umount lustre-OST0000 complete [13972.150309] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:41:20 (1786088480) [13980.076491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13980.542391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13984.921268] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [13985.762696] LustreError: 448141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13985.785883] LustreError: 448141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13989.858986] LustreError: 448142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13993.489989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13993.973815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13999.173787] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14006.536377] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14011.498861] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14029.991025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14030.179144] Lustre: Found index 0 for lustre-OST0000, updating log [14030.387980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14035.938808] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14043.733383] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14050.953835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14060.517361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14060.519658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14060.554136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14065.953106] Lustre: server umount lustre-OST0000 complete [14083.044279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14083.050049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14083.060110] Lustre: Skipped 2 previous similar messages [14083.067598] Lustre: Skipped 4 previous similar messages [14088.009363] Lustre: server umount lustre-MDT0000 complete [14088.162471] LustreError: 449876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14088.200237] LustreError: 449876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14090.208143] LNet: 1 local NIs in recovery (showing 1): 192.168.202.127@tcp [14092.004253] LustreError: 448122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786088602 with bad export cookie 4125054480377661292 [14092.014283] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14092.019740] LustreError: 448122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14092.627168] Lustre: server umount lustre-MDT0001 complete [14101.343845] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:43:29 (1786088609) [14723.077753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14723.420340] LustreError: 454572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14723.440867] LustreError: 454572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14723.483727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14727.550985] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14728.684143] LustreError: 454573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14732.769440] LustreError: 454572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14735.644344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14735.932220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14739.931774] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14747.822513] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14753.374832] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14773.487481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14773.763137] Lustre: Found index 0 for lustre-OST0000, updating log [14774.127873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14780.095423] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [14782.432749] LNet: 1 local NIs in recovery (showing 1): 192.168.202.127@tcp [14787.604668] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14789.461155] Lustre: server umount lustre-OST0000 complete [14794.597584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14794.615749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14794.621592] Lustre: Skipped 1 previous similar message [14795.683680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14795.696231] Lustre: Skipped 1 previous similar message [14796.707533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14796.717104] Lustre: Skipped 1 previous similar message [14798.761323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14798.764080] Lustre: Skipped 3 previous similar messages [14799.873306] Lustre: server umount lustre-MDT0000 complete [14800.806606] LustreError: 454572:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14800.819059] LustreError: 454572:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14803.762952] LustreError: 454552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786089314 with bad export cookie 4125054480377662377 [14803.779684] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14803.783602] LustreError: 454552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14804.044307] Lustre: server umount lustre-MDT0001 complete [14839.073839] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [14845.919560] LNet: 1 local NIs in recovery (showing 1): 192.168.202.127@tcp [14850.858422] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [14889.992345] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:56:38 (1786089398) [14916.902530] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [14922.039991] Key type lgssc unregistered [14922.626158] LNet: 461208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14922.653937] LNetError: 461208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15034.282854] LNet: Removed LNI 192.168.202.127@tcp [15035.261338] Key type .llcrypt unregistered [15035.265226] Key type ._llcrypt unregistered [15053.218352] Key type ._llcrypt registered [15053.220866] Key type .llcrypt registered [15053.325720] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [15054.711303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15054.869828] alg: No test for adler32 (adler32-zlib) [15055.942506] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [15056.111940] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [15057.743183] Key type lgssc registered [15059.135747] Lustre: Echo OBD driver; http://www.lustre.org/ [15063.423645] LNet: Added LNI 192.168.202.127@tcp100 [8/256/0/180] [15064.351032] LNet: Added LNI 192.168.202.127@tcp101 [8/256/0/180] [15065.487689] LNet: Added LNI 192.168.202.127@tcp102 [8/256/0/180] [15066.512084] LNet: Added LNI 192.168.202.127@tcp103 [8/256/0/180] [15067.595992] LNet: Added LNI 192.168.202.127@tcp104 [8/256/0/180] [15068.707414] LNet: Added LNI 192.168.202.127@tcp105 [8/256/0/180] [15069.661396] LNet: Added LNI 192.168.202.127@tcp106 [8/256/0/180] [15070.711368] LNet: Added LNI 192.168.202.127@tcp107 [8/256/0/180] [15071.814804] LNet: Added LNI 192.168.202.127@tcp108 [8/256/0/180] [15072.790641] LNet: Added LNI 192.168.202.127@tcp109 [8/256/0/180] [15073.738503] LNet: Added LNI 192.168.202.127@tcp110 [8/256/0/180] [15074.632735] LNet: Added LNI 192.168.202.127@tcp111 [8/256/0/180] [15076.076551] LNet: Added LNI 192.168.202.127@tcp112 [8/256/0/180] [15077.031078] LNet: Added LNI 192.168.202.127@tcp113 [8/256/0/180] [15077.984032] LNet: Added LNI 192.168.202.127@tcp114 [8/256/0/180] [15079.074421] LNet: Added LNI 192.168.202.127@tcp115 [8/256/0/180] [15080.198286] LNet: Added LNI 192.168.202.127@tcp116 [8/256/0/180] [15081.186729] LNet: Added LNI 192.168.202.127@tcp117 [8/256/0/180] [15082.178406] LNet: Added LNI 192.168.202.127@tcp118 [8/256/0/180] [15083.208606] LNet: Added LNI 192.168.202.127@tcp119 [8/256/0/180] [15084.161493] LNet: Added LNI 192.168.202.127@tcp120 [8/256/0/180] [15085.152675] LNet: Added LNI 192.168.202.127@tcp121 [8/256/0/180] [15086.218482] LNet: Added LNI 192.168.202.127@tcp122 [8/256/0/180] [15087.239050] LNet: Added LNI 192.168.202.127@tcp123 [8/256/0/180] [15088.355670] LNet: Added LNI 192.168.202.127@tcp124 [8/256/0/180] [15089.241945] LNet: Added LNI 192.168.202.127@tcp125 [8/256/0/180] [15090.257047] LNet: Added LNI 192.168.202.127@tcp126 [8/256/0/180] [15091.174165] LNet: Added LNI 192.168.202.127@tcp127 [8/256/0/180] [15092.101793] LNet: Added LNI 192.168.202.127@tcp128 [8/256/0/180] [15093.169312] LNet: Added LNI 192.168.202.127@tcp129 [8/256/0/180] [15094.104663] LNet: Added LNI 192.168.202.127@tcp130 [8/256/0/180] [15095.095856] LNet: Added LNI 192.168.202.127@tcp131 [8/256/0/180] [15096.032911] LNet: Added LNI 192.168.202.127@tcp132 [8/256/0/180] [15096.887966] LNet: Added LNI 192.168.202.127@tcp133 [8/256/0/180] [15097.474551] LNet: Added LNI 192.168.202.127@tcp134 [8/256/0/180] [15098.359027] LNet: Added LNI 192.168.202.127@tcp135 [8/256/0/180] [15105.562108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15105.586525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15106.760294] Lustre: 464237:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [15106.890330] LNetError: 461697:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [15106.911971] LNetError: Protocol error connecting to 192.168.202.127@tcp116 at host 192.168.202.127:988: is it running a compatible version of Lustre? [15106.932857] LNetError: Skipped 1 previous similar message [15107.052975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15107.100956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15107.222747] Lustre: lustre-MDT0000: new disk, initializing [15107.300151] LNet: 9 local NIs in recovery (showing 9): 192.168.202.127@tcp121, 192.168.202.127@tcp122, 192.168.202.127@tcp124, 192.168.202.127@tcp125, 192.168.202.127@tcp126, 192.168.202.127@tcp127, 192.168.202.127@tcp123, 192.168.202.127@tcp129, 192.168.202.127@tcp128 [15107.339719] Lustre: 464293:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [15107.367684] Lustre: 464293:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15107.388702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15107.409491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15109.408282] LNetError: 461696:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [15109.427296] LNetError: 461696:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 25 previous similar messages [15109.438200] LNetError: Protocol error connecting to 192.168.202.127@tcp121 at host 192.168.202.127:988: is it running a compatible version of Lustre? [15109.459429] LNetError: Skipped 25 previous similar messages [15110.495948] LNetError: 461697:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [15110.514395] LNetError: 461697:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [15110.541997] LNetError: Protocol error connecting to 192.168.202.127@tcp129 at host 192.168.202.127:988: is it running a compatible version of Lustre? [15112.906579] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15113.517038] LNetError: 461699:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [15113.528888] LNetError: Protocol error connecting to 192.168.202.127@tcp121 at host 192.168.202.127:988: is it running a compatible version of Lustre? [15121.715800] LNetError: 464258:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [15121.731240] LNetError: Protocol error connecting to 192.168.202.127@tcp121 at host 192.168.202.127:988: is it running a compatible version of Lustre? [15125.869508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15126.105496] Lustre: 464255:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15126.167626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15126.171796] Lustre: Skipped 1 previous similar message [15126.232927] Lustre: lustre-MDT0001: new disk, initializing [15126.266690] Lustre: 465054:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [15126.272886] Lustre: 465054:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15126.378802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15126.410646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15126.422529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15132.521382] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15133.805311] Lustre: 465372:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [15133.837948] Lustre: 465372:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15137.058586] LNetError: 461698:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [15137.079632] LNetError: Protocol error connecting to 192.168.202.127@tcp121 at host 192.168.202.127:988: is it running a compatible version of Lustre? [15142.485856] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15148.552736] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15154.400312] LNetError: 461699:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [15154.413417] LNetError: 461699:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 18 previous similar messages [15154.431933] LNetError: Protocol error connecting to 192.168.202.127@tcp125 at host 192.168.202.127:988: is it running a compatible version of Lustre? [15154.443542] LNetError: Skipped 18 previous similar messages [15155.418222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15155.886935] Lustre: lustre-OST0000: new disk, initializing [15155.895583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15155.911283] Lustre: 466239:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15156.056534] Lustre: 466244:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [15156.073378] Lustre: 466244:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15156.118949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15161.947391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15161.970427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15162.084812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15163.045915] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15166.751282] LNet: 1 local NIs in recovery (showing 1): 192.168.202.127@tcp121 [15172.186783] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15182.817088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15182.817116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15182.820270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15182.833333] Lustre: Skipped 1 previous similar message [15184.680885] Lustre: server umount lustre-OST0000 complete [15189.222894] LNetError: 461696:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [15189.243740] LNetError: 461696:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 29 previous similar messages [15189.269510] LNetError: Protocol error connecting to 192.168.202.127@tcp106 at host 192.168.202.127:988: is it running a compatible version of Lustre? [15189.292541] LNetError: Skipped 29 previous similar messages [15193.057044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15193.058985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15193.081850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15193.081858] Lustre: Skipped 1 previous similar message [15193.112669] LustreError: Skipped 1 previous similar message [15195.982486] Lustre: server umount lustre-MDT0000 complete [15200.168237] LustreError: 464246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786089710 with bad export cookie 320515177264635603 [15200.169162] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15200.201442] LustreError: 464246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15201.022327] Lustre: server umount lustre-MDT0001 complete [15215.315046] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [15219.811922] Key type lgssc unregistered [15220.413515] LNet: 468154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15220.429286] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15220.436298] LNet: Removed LNI 192.168.202.127@tcp [15221.471210] LNet: Removed LNI 192.168.202.127@tcp100 [15221.478422] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15221.493170] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15221.503799] LNet: Removed LNI 192.168.202.127@tcp101 [15222.559227] LNet: Removed LNI 192.168.202.127@tcp102 [15222.561347] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15222.583449] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15222.603983] LNet: Removed LNI 192.168.202.127@tcp103 [15223.712039] LNet: Removed LNI 192.168.202.127@tcp104 [15224.735198] LNet: Removed LNI 192.168.202.127@tcp105 [15224.740459] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15224.757303] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15224.768379] LNet: Removed LNI 192.168.202.127@tcp106 [15225.824686] LNet: Removed LNI 192.168.202.127@tcp107 [15226.847536] LNet: Removed LNI 192.168.202.127@tcp108 [15227.872235] LNet: Removed LNI 192.168.202.127@tcp109 [15228.895172] LNet: Removed LNI 192.168.202.127@tcp110 [15228.917154] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15228.925788] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15229.986043] LNet: Removed LNI 192.168.202.127@tcp111 [15231.007199] LNet: Removed LNI 192.168.202.127@tcp112 [15232.033137] LNet: Removed LNI 192.168.202.127@tcp113 [15233.055399] LNet: Removed LNI 192.168.202.127@tcp114 [15234.079318] LNet: Removed LNI 192.168.202.127@tcp115 [15235.103197] LNet: Removed LNI 192.168.202.127@tcp116 [15236.128703] LNet: Removed LNI 192.168.202.127@tcp117 [15237.151269] LNet: Removed LNI 192.168.202.127@tcp118 [15237.159031] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15237.173710] LNetError: 468154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [15237.189969] LNet: Removed LNI 192.168.202.127@tcp119 [15238.239186] LNet: Removed LNI 192.168.202.127@tcp120 [15238.242964] LNet: Removed LNI 192.168.202.127@tcp121 [15239.263874] LNet: Removed LNI 192.168.202.127@tcp122 [15240.287249] LNet: Removed LNI 192.168.202.127@tcp123 [15241.375238] LNet: Removed LNI 192.168.202.127@tcp124 [15242.399176] LNet: Removed LNI 192.168.202.127@tcp125 [15243.423845] LNet: Removed LNI 192.168.202.127@tcp126 [15244.447291] LNet: Removed LNI 192.168.202.127@tcp127 [15245.471458] LNet: Removed LNI 192.168.202.127@tcp128 [15246.496069] LNet: Removed LNI 192.168.202.127@tcp129 [15246.498075] LNet: Removed LNI 192.168.202.127@tcp130 [15246.517195] LNet: Removed LNI 192.168.202.127@tcp131 [15246.519223] LNet: Removed LNI 192.168.202.127@tcp132 [15246.547459] LNet: Removed LNI 192.168.202.127@tcp133 [15246.563093] LNet: Removed LNI 192.168.202.127@tcp134 [15246.589413] LNet: Removed LNI 192.168.202.127@tcp135 [15247.913160] Key type .llcrypt unregistered [15247.917161] Key type ._llcrypt unregistered [15275.608970] Key type ._llcrypt registered [15275.611316] Key type .llcrypt registered [15275.734605] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [15291.689949] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [15292.710902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15292.753257] alg: No test for adler32 (adler32-zlib) [15293.855795] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [15294.203946] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [15295.903187] Key type lgssc registered [15297.406946] Lustre: Echo OBD driver; http://www.lustre.org/ [15340.211468] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:04:08 (1786089848) [15362.022840] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [15365.578213] Key type lgssc unregistered [15365.901079] LNet: 472246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15365.919212] LNetError: 472246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15365.946235] LNet: Removed LNI 192.168.202.127@tcp [15367.089939] Key type .llcrypt unregistered [15367.092136] Key type ._llcrypt unregistered [15385.466283] Key type ._llcrypt registered [15385.468651] Key type .llcrypt registered [15385.592682] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [15387.078290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15387.110246] alg: No test for adler32 (adler32-zlib) [15388.312645] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [15388.601638] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [15390.287211] Key type lgssc registered [15391.717103] Lustre: Echo OBD driver; http://www.lustre.org/ [15402.155308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15402.176344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15403.525551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15403.551494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15403.614916] Lustre: lustre-MDT0000: new disk, initializing [15403.676175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15403.698449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15408.987708] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15420.486110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15420.631732] Lustre: 473457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15420.685265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15420.688670] Lustre: Skipped 1 previous similar message [15420.747949] Lustre: lustre-MDT0001: new disk, initializing [15420.834894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15420.877308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15420.888410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15425.924351] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15436.082297] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15441.790880] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15449.624214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15450.038821] Lustre: lustre-OST0000: new disk, initializing [15450.042334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15450.049691] Lustre: 475427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15450.187960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15457.402889] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15457.854607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15457.868669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15457.993226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15468.679187] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15474.665529] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing lnet_if_list [15482.647970] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15482.680349] LNet: Added LNI 192.168.202.127@tcp10 [8/256/0/180] [15489.845563] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15489.878055] LNet: Added LNI 192.168.202.127@tcp11 [8/256/0/180] [15496.143890] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15496.181387] LNet: Added LNI 192.168.202.127@tcp12 [8/256/0/180] [15500.727734] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15500.769154] LNet: Added LNI 192.168.202.127@tcp13 [8/256/0/180] [15506.639911] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15506.674873] LNet: Added LNI 192.168.202.127@tcp14 [8/256/0/180] [15512.024855] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15512.079734] LNet: Added LNI 192.168.202.127@tcp15 [8/256/0/180] [15517.067683] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15517.099902] LNet: Added LNI 192.168.202.127@tcp16 [8/256/0/180] [15523.447666] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15523.498757] LNet: Added LNI 192.168.202.127@tcp17 [8/256/0/180] [15529.162022] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15529.197337] LNet: Added LNI 192.168.202.127@tcp18 [8/256/0/180] [15535.593993] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15535.619338] LNet: Added LNI 192.168.202.127@tcp19 [8/256/0/180] [15541.736846] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15541.792910] LNet: Added LNI 192.168.202.127@tcp20 [8/256/0/180] [15548.111804] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15548.148866] LNet: Added LNI 192.168.202.127@tcp21 [8/256/0/180] [15555.050499] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15555.085774] LNet: Added LNI 192.168.202.127@tcp22 [8/256/0/180] [15563.734719] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15563.759241] LNet: Added LNI 192.168.202.127@tcp23 [8/256/0/180] [15570.081922] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15570.125395] LNet: Added LNI 192.168.202.127@tcp24 [8/256/0/180] [15576.791253] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15576.819981] LNet: Added LNI 192.168.202.127@tcp25 [8/256/0/180] [15583.385820] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15583.417933] LNet: Added LNI 192.168.202.127@tcp26 [8/256/0/180] [15590.501770] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15590.539719] LNet: Added LNI 192.168.202.127@tcp27 [8/256/0/180] [15597.430542] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15597.478415] LNet: Added LNI 192.168.202.127@tcp28 [8/256/0/180] [15604.551589] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15604.573379] LNet: Added LNI 192.168.202.127@tcp29 [8/256/0/180] [15611.864345] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15611.899063] LNet: Added LNI 192.168.202.127@tcp30 [8/256/0/180] [15619.027293] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15619.056294] LNet: Added LNI 192.168.202.127@tcp31 [8/256/0/180] [15625.330789] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15625.353722] LNet: Added LNI 192.168.202.127@tcp32 [8/256/0/180] [15631.784297] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15631.818576] LNet: Added LNI 192.168.202.127@tcp33 [8/256/0/180] [15637.384181] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15637.418442] LNet: Added LNI 192.168.202.127@tcp34 [8/256/0/180] [15644.144887] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15644.189161] LNet: Added LNI 192.168.202.127@tcp35 [8/256/0/180] [15651.528318] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15651.569733] LNet: Added LNI 192.168.202.127@tcp36 [8/256/0/180] [15658.087388] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15658.113263] LNet: Added LNI 192.168.202.127@tcp37 [8/256/0/180] [15665.401411] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15665.458284] LNet: Added LNI 192.168.202.127@tcp38 [8/256/0/180] [15673.829736] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15673.853948] LNet: Added LNI 192.168.202.127@tcp39 [8/256/0/180] [15681.643470] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15681.680388] LNet: Added LNI 192.168.202.127@tcp40 [8/256/0/180] [15690.203948] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15690.251457] LNet: Added LNI 192.168.202.127@tcp41 [8/256/0/180] [15699.179963] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15699.214440] LNet: Added LNI 192.168.202.127@tcp42 [8/256/0/180] [15707.431922] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15707.487367] LNet: Added LNI 192.168.202.127@tcp43 [8/256/0/180] [15716.701556] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15716.753733] LNet: Added LNI 192.168.202.127@tcp44 [8/256/0/180] [15724.625981] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15724.670096] LNet: Added LNI 192.168.202.127@tcp45 [8/256/0/180] [15733.181683] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15733.210834] LNet: Added LNI 192.168.202.127@tcp46 [8/256/0/180] [15741.960854] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15742.004288] LNet: Added LNI 192.168.202.127@tcp47 [8/256/0/180] [15751.551859] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15751.578304] LNet: Added LNI 192.168.202.127@tcp48 [8/256/0/180] [15760.106391] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15760.146963] LNet: Added LNI 192.168.202.127@tcp49 [8/256/0/180] [15768.743135] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15768.767648] LNet: Added LNI 192.168.202.127@tcp50 [8/256/0/180] [15789.537305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15789.556996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15789.571300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15791.083269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15791.087726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15791.099102] Lustre: Skipped 1 previous similar message [15794.744549] Lustre: server umount lustre-MDT0000 complete [15798.496629] LustreError: 480410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786090308 with bad export cookie 1650610219429979217 [15798.505815] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15798.510041] LustreError: 480410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15798.935398] Lustre: server umount lustre-MDT0001 complete [15816.676642] Lustre: 472732:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786090311/real 1786090311] req@000000000f6466cd x1872851002119040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786090327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15816.699822] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15818.783149] Lustre: 472732:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786090313/real 1786090313] req@00000000d24248fb x1872851002119168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786090329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15819.824959] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [15821.795072] Lustre: 472731:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786090316/real 1786090316] req@000000004f6a3277 x1872851002119296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786090332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15824.931409] Lustre: 472731:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786090319/real 1786090319] req@000000004f3985ce x1872851002119552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786090335 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15829.156137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15831.008389] Lustre: 472733:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786090325/real 1786090325] req@0000000084e5f687 x1872851002119680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786090341 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15844.830613] LustreError: 475421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15844.856789] LustreError: 475421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15844.902961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15845.921264] LustreError: 475422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15845.934980] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15847.971334] LustreError: 475421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15849.490913] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15853.031843] LustreError: 475422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15857.375768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15857.689348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15861.797827] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15861.994491] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [15869.574583] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15874.840263] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15896.046210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15896.058537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15896.092279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15896.107564] Lustre: Skipped 2 previous similar messages [15900.863257] Lustre: server umount lustre-OST0000 complete [15907.821649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15907.830937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15907.833246] Lustre: Skipped 2 previous similar messages [15907.851218] Lustre: Skipped 3 previous similar messages [15911.314684] Lustre: server umount lustre-MDT0000 complete [15912.929285] LustreError: 499676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15912.965642] LustreError: 499676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15915.526818] LustreError: 473450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786090425 with bad export cookie 1650610219429981912 [15915.534383] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15915.548752] LustreError: 473450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15916.034869] Lustre: server umount lustre-MDT0001 complete [15928.418782] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [15931.816744] Key type lgssc unregistered [15932.368355] LNet: 502473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15932.385948] LNetError: 502473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15932.397784] LNet: Removed LNI 192.168.202.127@tcp [15932.408313] LNet: Removed LNI 192.168.202.127@tcp10 [15932.411782] LNet: Removed LNI 192.168.202.127@tcp11 [15932.418329] LNet: Removed LNI 192.168.202.127@tcp12 [15932.429356] LNet: Removed LNI 192.168.202.127@tcp13 [15932.436263] LNet: Removed LNI 192.168.202.127@tcp14 [15932.439344] LNet: Removed LNI 192.168.202.127@tcp15 [15932.443566] LNet: Removed LNI 192.168.202.127@tcp16 [15932.446799] LNet: Removed LNI 192.168.202.127@tcp17 [15932.453116] LNet: Removed LNI 192.168.202.127@tcp18 [15932.458776] LNet: Removed LNI 192.168.202.127@tcp19 [15932.464376] LNet: Removed LNI 192.168.202.127@tcp20 [15932.467749] LNet: Removed LNI 192.168.202.127@tcp21 [15932.471533] LNet: Removed LNI 192.168.202.127@tcp22 [15932.474553] LNet: Removed LNI 192.168.202.127@tcp23 [15932.477133] LNet: Removed LNI 192.168.202.127@tcp24 [15932.480469] LNet: Removed LNI 192.168.202.127@tcp25 [15932.483814] LNet: Removed LNI 192.168.202.127@tcp26 [15932.486235] LNet: Removed LNI 192.168.202.127@tcp27 [15932.489761] LNet: Removed LNI 192.168.202.127@tcp28 [15932.492961] LNet: Removed LNI 192.168.202.127@tcp29 [15932.495509] LNet: Removed LNI 192.168.202.127@tcp30 [15932.498279] LNet: Removed LNI 192.168.202.127@tcp31 [15932.501146] LNet: Removed LNI 192.168.202.127@tcp32 [15932.503374] LNet: Removed LNI 192.168.202.127@tcp33 [15932.506206] LNet: Removed LNI 192.168.202.127@tcp34 [15932.508526] LNet: Removed LNI 192.168.202.127@tcp35 [15932.512756] LNet: Removed LNI 192.168.202.127@tcp36 [15932.515740] LNet: Removed LNI 192.168.202.127@tcp37 [15932.517718] LNet: Removed LNI 192.168.202.127@tcp38 [15932.519556] LNet: Removed LNI 192.168.202.127@tcp39 [15932.525648] LNet: Removed LNI 192.168.202.127@tcp40 [15932.532295] LNet: Removed LNI 192.168.202.127@tcp41 [15932.535265] LNet: Removed LNI 192.168.202.127@tcp42 [15932.537583] LNet: Removed LNI 192.168.202.127@tcp43 [15932.540116] LNet: Removed LNI 192.168.202.127@tcp44 [15932.542354] LNet: Removed LNI 192.168.202.127@tcp45 [15932.544856] LNet: Removed LNI 192.168.202.127@tcp46 [15932.547441] LNet: Removed LNI 192.168.202.127@tcp47 [15932.550776] LNet: Removed LNI 192.168.202.127@tcp48 [15932.555191] LNet: Removed LNI 192.168.202.127@tcp49 [15932.564698] LNet: Removed LNI 192.168.202.127@tcp50 [15933.280697] Key type .llcrypt unregistered [15933.282377] Key type ._llcrypt unregistered [15955.236086] Key type ._llcrypt registered [15955.238730] Key type .llcrypt registered [15955.378266] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [15971.251332] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [15972.725523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15972.761029] alg: No test for adler32 (adler32-zlib) [15973.781096] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [15974.050098] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [15975.735215] Key type lgssc registered [15977.260823] Lustre: Echo OBD driver; http://www.lustre.org/ [16014.205231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16014.244578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16015.695653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16015.744456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16015.846778] Lustre: lustre-MDT0000: new disk, initializing [16015.914524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16015.927928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16021.062555] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16032.266876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16032.409882] Lustre: 505672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16032.469911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16032.476282] Lustre: Skipped 1 previous similar message [16032.541271] Lustre: lustre-MDT0001: new disk, initializing [16032.627908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16032.669254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16032.680456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16038.169494] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16048.644390] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16056.903348] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16069.598378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16070.014184] Lustre: *** cfs_fail_loc=90f, val=0*** [16070.237895] Lustre: lustre-OST0000: new disk, initializing [16070.244707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16070.262987] Lustre: 507690:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16070.411819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16076.953532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16076.976262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16077.267064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16078.465406] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16094.094737] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16108.004062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16108.006303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16108.028826] Lustre: Skipped 1 previous similar message [16108.035407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16108.044189] Lustre: Skipped 1 previous similar message [16109.559497] Lustre: server umount lustre-OST0000 complete [16114.658495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16114.665777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16114.682304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16116.204560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16116.213155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16120.197281] Lustre: server umount lustre-MDT0000 complete [16125.065684] LustreError: 505665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786090635 with bad export cookie 4617016025568977977 [16125.069058] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16125.078058] LustreError: 505665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16125.824146] Lustre: server umount lustre-MDT0001 complete [16140.166733] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [16144.403822] Key type lgssc unregistered [16144.841482] LNet: 509597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16144.846457] LNetError: 509597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16144.867413] LNet: Removed LNI 192.168.202.127@tcp [16146.273288] Key type .llcrypt unregistered [16146.276454] Key type ._llcrypt unregistered [16167.010198] Key type ._llcrypt registered [16167.015385] Key type .llcrypt registered [16167.131951] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [16168.786227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16168.846729] alg: No test for adler32 (adler32-zlib) [16169.964357] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [16170.319662] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [16172.119241] Key type lgssc registered [16173.983969] Lustre: Echo OBD driver; http://www.lustre.org/ [16185.538311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16185.551601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16187.200101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16193.520304] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16196.585157] LustreError: 510867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16196.600727] LustreError: 510867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16201.718892] LustreError: 510868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16204.296135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16204.925391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16210.542498] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16221.039403] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16227.323500] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16235.103619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16235.463102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16242.316303] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16251.244431] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16266.917344] loop0: detected capacity change from 0 to 204800000 [16267.156687] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16267.185681] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16267.226312] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16273.656824] loop0: detected capacity change from 0 to 204800000 [16273.962732] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16273.978861] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16274.008764] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16280.133816] loop0: detected capacity change from 0 to 204800000 [16286.914135] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16287.125476] Lustre: fs15246-OST0000: new disk, initializing [16287.131172] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16287.142563] Lustre: 514463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16287.204197] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16294.831789] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16306.333688] loop1: detected capacity change from 0 to 204800000 [16313.300366] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16313.459981] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16313.526698] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16313.717503] Lustre: fs15246-MDT0000: new disk, initializing [16313.892795] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16313.912498] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16314.041560] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16315.108159] Lustre: 515589:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000006fa84f4 x1872851820921984/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786090841 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16319.170993] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16324.194359] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16324.210297] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16324.240253] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16342.500332] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16342.521849] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16347.619159] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16348.138926] Lustre: server umount fs15246-MDT0000 complete [16352.740048] LustreError: 510863:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16357.872040] LustreError: 510862:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16360.605610] Lustre: server umount fs15246-OST0000 complete [16377.828524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16377.849281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16377.886503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16383.457660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16383.480916] Lustre: Skipped 3 previous similar messages [16388.585053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16388.604058] Lustre: Skipped 1 previous similar message [16391.139554] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16391.262538] Lustre: server umount lustre-OST0000 complete [16398.816974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16398.824614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16398.830083] Lustre: Skipped 2 previous similar messages [16402.676378] Lustre: server umount lustre-MDT0000 complete [16403.948581] LustreError: 510863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16403.986581] LustreError: 510863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16406.914358] LustreError: 510849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786090917 with bad export cookie 162946105387422793 [16406.917447] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16406.929270] LustreError: 510849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16407.434699] Lustre: server umount lustre-MDT0001 complete [16418.685513] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [16421.563698] Key type lgssc unregistered [16421.882763] LNet: 518691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16421.893864] LNetError: 518691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16421.914870] LNet: Removed LNI 192.168.202.127@tcp [16423.019909] Key type .llcrypt unregistered [16423.021880] Key type ._llcrypt unregistered [16443.255289] Key type ._llcrypt registered [16443.257331] Key type .llcrypt registered [16443.354510] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [16444.633097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16444.716081] alg: No test for adler32 (adler32-zlib) [16445.989673] Lustre: Lustre: Build Version: 2.17.56_3_g28608f2 [16446.314417] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [16448.027045] Key type lgssc registered [16449.251657] Lustre: Echo OBD driver; http://www.lustre.org/ [16489.945938] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [16500.162732] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [16541.096780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16541.145754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16542.534128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16542.614662] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16542.708260] Lustre: lustre-MDT0000: new disk, initializing [16542.792494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16542.808697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16547.114381] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16558.134916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16558.282263] Lustre: 523161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16558.344338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16558.347896] Lustre: Skipped 1 previous similar message [16558.452310] Lustre: lustre-MDT0001: new disk, initializing [16558.569300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16558.619646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16558.634667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16564.224733] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16574.972220] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16581.229832] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16589.475066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16589.946971] Lustre: lustre-OST0000: new disk, initializing [16589.963928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16589.978934] Lustre: 525128:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16590.100839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16596.648665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16596.668595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16596.856455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16597.648772] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [16609.954817] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16617.628184] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16618.126888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16623.785922] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16624.118286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16627.683643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16627.692420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16627.694446] LustreError: Skipped 1 previous similar message [16627.696958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16627.743804] Lustre: Skipped 1 previous similar message [16632.593332] Lustre: server umount lustre-OST0000 complete [16637.136066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16637.142598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16637.157450] Lustre: Skipped 2 previous similar messages [16642.530220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16642.545952] Lustre: Skipped 1 previous similar message [16643.657643] Lustre: server umount lustre-MDT0000 complete [16647.523204] LustreError: 523152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786091157 with bad export cookie 4140994918412549549 [16647.529499] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16647.536885] LustreError: 523152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16647.876801] Lustre: server umount lustre-MDT0001 complete [16674.089947] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [16688.236984] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [16727.510585] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16419 sec ========= 04:27:15 (1786091235) [16730.139745] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:27:17 (1786091237) === [16734.624672] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:27:22 (1786091242) === [16787.687274] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [16790.242973] Key type lgssc unregistered [16790.561465] LNet: 531777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16790.568497] LNetError: 531777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16791.591173] LNet: Removed LNI 192.168.202.127@tcp [16792.514580] Key type .llcrypt unregistered [16792.517096] Key type ._llcrypt unregistered