[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 458602474 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003180] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008510] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011159] LSM: Security Framework initializing [ 0.012080] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.014093] *** VALIDATE selinux *** [ 0.022595] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027109] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] *** VALIDATE tmpfs *** [ 0.031424] *** VALIDATE proc *** [ 0.032235] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041446] debug: unmapping init [mem 0xffffffff9d659000-0xffffffff9d660fff] [ 0.043203] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044713] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.045991] ... bit width: 48 [ 0.046016] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.049972] ... event mask: 000000070000000f [ 0.050300] rcu: Hierarchical SRCU implementation. [ 0.052601] smp: Bringing up secondary CPUs ... [ 0.053589] x86: Booting SMP configuration: [ 0.054021] .... node #0, CPUs: #1 #2 #3 [ 0.057448] smp: Brought up 1 node, 4 CPUs [ 0.059013] smpboot: Max logical packages: 1 [ 0.060016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125296] node 0 deferred pages initialised in 61ms [ 0.129032] devtmpfs: initialized [ 0.130230] x86/mm: Memory block size: 128MB [ 0.132647] gcov: version magic: 0x41383552 [ 0.135203] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.137078] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139453] pinctrl core: initialized pinctrl subsystem [ 0.140201] [ 0.140581] ************************************************************* [ 0.143010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144008] ** ** [ 0.146009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147008] ** ** [ 0.149009] ** This means that this kernel is built to expose internal ** [ 0.150010] ** IOMMU data structures, which may compromise security on ** [ 0.152010] ** your system. ** [ 0.153008] ** ** [ 0.155012] ** If you see this message and you are not debugging the ** [ 0.156009] ** kernel, report this immediately to your vendor! ** [ 0.158010] ** ** [ 0.160008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161007] ************************************************************* [ 0.163678] NET: Registered protocol family 16 [ 0.164492] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177046] cpuidle: using governor menu [ 0.178570] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179420] PCI: Using configuration type 1 for base access [ 0.180149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186049] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192087] cryptd: max_cpu_qlen set to 1000 [ 0.195330] ACPI: Added _OSI(Module Device) [ 0.198015] ACPI: Added _OSI(Processor Device) [ 0.199012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.206358] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.212621] ACPI: Interpreter enabled [ 0.214059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.216013] ACPI: Using IOAPIC for interrupt routing [ 0.217092] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.219400] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228813] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.230025] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.232013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.233065] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236977] acpiphp: Slot [2] registered [ 0.238068] acpiphp: Slot [5] registered [ 0.239000] acpiphp: Slot [6] registered [ 0.241114] acpiphp: Slot [7] registered [ 0.242117] acpiphp: Slot [8] registered [ 0.244086] acpiphp: Slot [9] registered [ 0.245121] acpiphp: Slot [10] registered [ 0.246129] acpiphp: Slot [3] registered [ 0.248095] acpiphp: Slot [4] registered [ 0.249115] acpiphp: Slot [11] registered [ 0.251096] acpiphp: Slot [12] registered [ 0.252094] acpiphp: Slot [13] registered [ 0.253094] acpiphp: Slot [14] registered [ 0.255066] acpiphp: Slot [15] registered [ 0.256072] acpiphp: Slot [16] registered [ 0.257066] acpiphp: Slot [17] registered [ 0.258109] acpiphp: Slot [18] registered [ 0.259066] acpiphp: Slot [19] registered [ 0.260043] acpiphp: Slot [20] registered [ 0.261107] acpiphp: Slot [21] registered [ 0.263121] acpiphp: Slot [22] registered [ 0.265109] acpiphp: Slot [23] registered [ 0.266212] acpiphp: Slot [24] registered [ 0.268128] acpiphp: Slot [25] registered [ 0.269101] acpiphp: Slot [26] registered [ 0.271115] acpiphp: Slot [27] registered [ 0.273099] acpiphp: Slot [28] registered [ 0.274120] acpiphp: Slot [29] registered [ 0.276107] acpiphp: Slot [30] registered [ 0.277133] acpiphp: Slot [31] registered [ 0.279060] PCI host bridge to bus 0000:00 [ 0.281025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.283023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.285024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.287025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.290022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.292023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.294228] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.298008] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.300126] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.309638] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313439] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.316020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.318018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.320019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.323491] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.326800] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.330042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.333207] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.337013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.348014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.356504] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.363015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.372019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.393021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.408867] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.419018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.430020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.440017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.453401] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.460019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.467017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.482035] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.492496] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.499015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.506021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.521020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.537152] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.543014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.548014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.565026] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.574586] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.581014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.590018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.614018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.627041] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.629405] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.631364] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.635499] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.637229] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.642134] iommu: Default domain type: Passthrough [ 0.643316] SCSI subsystem initialized [ 0.645167] ACPI: bus type USB registered [ 0.646097] usbcore: registered new interface driver usbfs [ 0.648088] usbcore: registered new interface driver hub [ 0.650082] usbcore: registered new device driver usb [ 0.652181] pps_core: LinuxPPS API ver. 1 registered [ 0.654011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.657071] PTP clock support registered [ 0.659100] EDAC MC: Ver: 3.0.0 [ 0.661138] PCI: Using ACPI for IRQ routing [ 0.663107] NetLabel: Initializing [ 0.663891] NetLabel: domain hash size = 128 [ 0.664009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.665061] NetLabel: unlabeled traffic allowed by default [ 0.667007] vgaarb: loaded [ 0.668214] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.669011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.676000] clocksource: Switched to clocksource kvm-clock [ 0.788703] VFS: Disk quotas dquot_6.6.0 [ 0.791132] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.793671] *** VALIDATE ramfs *** [ 0.794689] *** VALIDATE hugetlbfs *** [ 0.796128] pnp: PnP ACPI init [ 0.798375] pnp: PnP ACPI: found 6 devices [ 0.821398] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.825208] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.827629] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.829992] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.832715] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.835519] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.838603] NET: Registered protocol family 2 [ 0.841600] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.847493] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.851771] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.856871] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.860734] TCP: Hash tables configured (established 65536 bind 65536) [ 0.864106] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.867515] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.870425] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.873406] NET: Registered protocol family 1 [ 0.876697] RPC: Registered named UNIX socket transport module. [ 0.878705] RPC: Registered udp transport module. [ 0.880459] RPC: Registered tcp transport module. [ 0.881920] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.884102] NET: Registered protocol family 44 [ 0.885977] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.888194] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.890393] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.893173] PCI: CLS 0 bytes, default 64 [ 0.895868] Unpacking initramfs... [ 2.303671] debug: unmapping init [mem 0xffff905d3cc54000-0xffff905d3ffbffff] [ 2.307527] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.309864] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.312815] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.757752] Initialise system trusted keyrings [ 2.758768] Key type blacklist registered [ 2.759886] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.769435] zbud: loaded [ 2.772407] *** VALIDATE nfs *** [ 2.773308] *** VALIDATE nfs4 *** [ 2.774533] pstore: using deflate compression [ 2.778388] Platform Keyring initialized [ 2.882997] NET: Registered protocol family 38 [ 2.884933] Key type asymmetric registered [ 2.886601] Asymmetric key parser 'x509' registered [ 2.888486] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.891741] io scheduler mq-deadline registered [ 2.893545] io scheduler kyber registered [ 2.895404] io scheduler bfq registered [ 2.897391] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.900683] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.903306] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.905370] ACPI: Power Button [PWRF] [ 2.910960] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.918057] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.929619] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.938527] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.952457] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.980949] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.009361] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.013922] Non-volatile memory driver v1.3 [ 3.015386] Linux agpgart interface v0.103 [ 3.048509] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.051762] vda: detected capacity change from 0 to 74698752 [ 3.078828] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.081282] vdb: detected capacity change from 0 to 1073741824 [ 3.098636] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.102241] vdc: detected capacity change from 0 to 2621440000 [ 3.124245] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.127488] vdd: detected capacity change from 0 to 2621440000 [ 3.143793] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.146807] vde: detected capacity change from 0 to 4294967296 [ 3.163661] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.167287] vdf: detected capacity change from 0 to 4294967296 [ 3.175292] libphy: Fixed MDIO Bus: probed [ 3.188522] usbcore: registered new interface driver usbserial_generic [ 3.191399] usbserial: USB Serial support registered for generic [ 3.194613] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.199501] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.201568] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.204393] mousedev: PS/2 mouse device common for all mice [ 3.207621] rtc_cmos 00:05: RTC can wake from S4 [ 3.209634] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.211000] rtc_cmos 00:05: registered as rtc0 [ 3.214935] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.219336] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.220303] intel_pstate: CPU model not supported [ 3.223258] hid: raw HID events driver (C) Jiri Kosina [ 3.228113] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.229288] usbcore: registered new interface driver usbhid [ 3.237228] usbhid: USB HID core driver [ 3.239230] drop_monitor: Initializing network drop monitor service [ 3.241906] Initializing XFRM netlink socket [ 3.244522] NET: Registered protocol family 10 [ 3.248385] Segment Routing with IPv6 [ 3.249870] NET: Registered protocol family 17 [ 3.251960] mpls_gso: MPLS GSO support [ 3.257603] RAS: Correctable Errors collector initialized. [ 3.259487] AVX version of gcm_enc/dec engaged. [ 3.261047] AES CTR mode by8 optimization enabled [ 3.345352] sched_clock: Marking stable (3345322930, 0)->(4201574190, -856251260) [ 3.348966] registered taskstats version 1 [ 3.351354] Loading compiled-in X.509 certificates [ 3.353566] zswap: loaded using pool lzo/zbud [ 3.378928] Key type big_key registered [ 3.391840] Key type encrypted registered [ 3.393442] ima: No TPM chip found, activating TPM-bypass! [ 3.394957] ima: Allocated hash algorithm: sha1 [ 3.396172] ima: No architecture policies found [ 3.397357] evm: Initialising EVM extended attributes: [ 3.398761] evm: security.selinux [ 3.400080] evm: security.ima [ 3.400647] evm: security.capability [ 3.401779] evm: HMAC attrs: 0x1 [ 3.403911] rtc_cmos 00:05: setting system clock to 2026-09-03 12:53:52 UTC (1788440032) [ 3.409419] debug: unmapping init [mem 0xffffffff9e603000-0xffffffff9e7fffff] [ 3.411696] debug: unmapping init [mem 0xffffffff9d382000-0xffffffff9d658fff] [ 3.420270] Write protecting the kernel read-only data: 28672k [ 3.423190] debug: unmapping init [mem 0xffffffff9ba03000-0xffffffff9bbfffff] [ 3.425734] debug: unmapping init [mem 0xffffffff9c314000-0xffffffff9c3fffff] [ 3.458902] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.465831] systemd[1]: Detected virtualization kvm. [ 3.468067] systemd[1]: Detected architecture x86-64. [ 3.470253] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.496179] systemd[1]: No hostname configured. [ 3.497881] systemd[1]: Set hostname to . [ 3.500063] random: systemd: uninitialized urandom read (16 bytes read) [ 3.502478] systemd[1]: Initializing machine ID from random generator. [ 3.551596] random: ln: uninitialized urandom read (6 bytes read) [ 3.661625] random: systemd: uninitialized urandom read (16 bytes read) [ 3.664139] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.668973] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.673318] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.201894] device-mapper: uevent: version 1.0.3 [ 4.204146] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager.[ 4.540882] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.954844] virtio_net virtio0 ens2: renamed from eth0 [ 5.039879] scsi host0: ata_piix [ 5.113725] scsi host1: ata_piix [ 5.115267] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.117803] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.246391] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.907608] random: crng init done [ 9.908746] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.183548] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.319691] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.548229] SELinux: Disabled at runtime. [ 11.608104] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.617679] systemd[1]: Detected virtualization kvm. [ 11.619758] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.066322] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.069647] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.074809] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.078903] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.082403] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.090758] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.098834] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.161563] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.518572] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.875829] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.916364] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.006107] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.029864] EDAC sbridge: Ver: 1.1.2 [ 14.225628] Key type dns_resolver registered [ 14.524265] NFS: Registering the id_resolver key type [ 14.526185] Key type id_resolver registered [ 14.527647] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg158-server login: [ 60.842155] hrtimer: interrupt took 3032619 ns [ 76.956206] libcfs: loading out-of-tree module taints kernel. [ 77.020901] Key type ._llcrypt registered [ 77.022940] Key type .llcrypt registered [ 77.170389] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 96.406523] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 98.183504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.207471] alg: No test for adler32 (adler32-zlib) [ 99.577886] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 100.517570] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 102.327194] Key type lgssc registered [ 104.079713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.513402] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 171.275490] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 185.956621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 185.991894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.368845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 187.420673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 187.569584] Lustre: lustre-MDT0000: new disk, initializing [ 187.696448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.721488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 192.535777] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.304714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.392154] Lustre: 6553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 206.423184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 206.428434] Lustre: Skipped 1 previous similar message [ 206.548785] Lustre: lustre-MDT0001: new disk, initializing [ 206.615989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 206.651924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 206.669729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.659524] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.667681] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 227.013397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 227.566983] Lustre: lustre-OST0000: new disk, initializing [ 227.570771] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 227.579542] Lustre: 8492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.688340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.847931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 228.858812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 228.989318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.256975] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.613455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.820447] Lustre: lustre-OST0001: new disk, initializing [ 252.826906] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 252.839416] Lustre: 9565:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.961638] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 259.837325] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.652187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 260.663924] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 260.767552] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 271.679733] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.368535] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 290.075817] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing check_logdir /tmp/testlogs/ [ 296.286182] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing yml_node [ 302.414250] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 304.999689] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 307.640335] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 310.002209] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:58:57 EDT 2026 [ 328.259434] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 330.430516] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 347.616146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.624489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.627604] Lustre: Skipped 1 previous similar message [ 352.470785] Lustre: server umount lustre-MDT0000 complete [ 356.873490] LustreError: 7968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440385 with bad export cookie 7543622699989771059 [ 356.892644] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 357.858093] LustreError: 6560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.861214] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.867768] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 357.867778] Lustre: Skipped 3 previous similar messages [ 357.891720] LustreError: 6560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 357.929549] Lustre: Skipped 3 previous similar messages [ 361.893382] LustreError: 7829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.897869] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 361.914045] LustreError: 7829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 361.926205] Lustre: Skipped 2 previous similar messages [ 363.282994] Lustre: server umount lustre-MDT0001 complete [ 367.575025] Lustre: server umount lustre-OST0000 complete [ 371.579479] Lustre: server umount lustre-OST0001 complete [ 378.889436] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 389.777325] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 432.427437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.608930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 432.642560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 432.694873] Lustre: lustre-MDT0000: new disk, initializing [ 432.765537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.795212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 437.977299] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 449.245895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.424194] Lustre: 15132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 449.508690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 449.512597] Lustre: Skipped 1 previous similar message [ 449.594455] Lustre: lustre-MDT0001: new disk, initializing [ 449.678116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 449.718032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 449.731904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 454.454060] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 464.533762] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 471.167683] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 478.050898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 478.400164] Lustre: lustre-OST0000: new disk, initializing [ 478.403081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 478.408468] Lustre: 17100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 478.525744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.643402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 479.655108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 479.776676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 484.470643] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 494.954694] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 499.629389] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 499.915386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 504.547335] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 504.791189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 510.439560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.450761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 510.469195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 512.790918] Lustre: server umount lustre-OST0000 complete [ 516.576252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 516.580634] LustreError: Skipped 1 previous similar message [ 516.592257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.606397] Lustre: Skipped 1 previous similar message [ 516.622371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.629313] Lustre: Skipped 1 previous similar message [ 520.677465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.763755] Lustre: server umount lustre-MDT0000 complete [ 525.798945] LustreError: 15941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.822347] LustreError: 15941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 526.338866] LustreError: 15123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440555 with bad export cookie 7543622699989772592 [ 526.350346] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 526.351454] LustreError: 15123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 526.677957] Lustre: server umount lustre-MDT0001 complete [ 530.984647] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:02:38 (1788440558) [ 533.162733] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 534.826708] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:02:42 (1788440562) [ 542.001926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.512721] LustreError: 19507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.592605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 546.945648] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 547.840459] LustreError: 19508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.300336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.332747] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 567.618593] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 573.967725] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 580.622063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 587.151242] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 594.503980] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 605.666996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 605.669188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.675408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 605.675416] Lustre: Skipped 2 previous similar messages [ 605.697195] Lustre: Skipped 1 previous similar message [ 610.323843] Lustre: server umount lustre-OST0000 complete [ 619.992813] Lustre: server umount lustre-MDT0000 complete [ 621.024728] LustreError: 19507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.050418] LustreError: 19507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 623.855060] LustreError: 19489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440652 with bad export cookie 7543622699989773467 [ 623.858526] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.872025] LustreError: 19489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 624.411797] Lustre: server umount lustre-MDT0001 complete [ 635.881533] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 639.141607] Key type lgssc unregistered [ 639.495858] LNet: 23040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.500736] LNetError: 23040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.526854] LNet: Removed LNI 192.168.201.158@tcp [ 640.595255] Key type .llcrypt unregistered [ 640.597824] Key type ._llcrypt unregistered [ 664.842214] Key type ._llcrypt registered [ 664.847503] Key type .llcrypt registered [ 664.931378] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 665.917929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 666.110796] alg: No test for adler32 (adler32-zlib) [ 667.126858] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 667.360943] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 669.039194] Key type lgssc registered [ 670.083971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 677.938070] loop: module loaded [ 678.042691] loop0: detected capacity change from 0 to 204800000 [ 678.370566] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.400089] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.689698] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.826143] loop0: detected capacity change from 0 to 204800000 [ 685.014924] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.028104] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.048374] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.146454] loop0: detected capacity change from 0 to 204800000 [ 696.133231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.163368] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.452948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 697.506235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 697.563748] Lustre: lustre-MDT0000: new disk, initializing [ 697.642744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 697.689527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 702.358767] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 711.289704] loop1: detected capacity change from 0 to 204800000 [ 717.129592] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 717.585677] Lustre: lustre-OST07c6: new disk, initializing [ 717.588797] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 717.595958] Lustre: Skipped 1 previous similar message [ 717.614046] Lustre: 26233:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 717.718815] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 723.897499] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 726.667272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 726.682409] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 726.724182] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 730.233802] Lustre: 25219:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 730.247057] Lustre: 25219:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 730.252272] Lustre: 25219:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 730.258636] Lustre: 25219:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 730.270452] Lustre: 25219:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 730.277415] Lustre: 25219:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 747.747116] Lustre: 25218:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 747.755721] Lustre: 25218:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 747.759661] Lustre: 25218:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 747.767444] Lustre: 25218:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 747.771530] Lustre: 25218:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 747.775211] Lustre: 25218:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 752.726572] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 752.742554] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 752.762499] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 752.772869] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.781636] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 752.787738] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.802816] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 752.823760] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.834305] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 752.860035] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.873868] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 752.886089] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.133340] Lustre: 25218:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 756.145732] Lustre: 25218:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 756.157712] Lustre: 25218:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 756.166137] Lustre: 25218:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.173758] Lustre: 25218:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 756.190648] Lustre: 25218:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.202437] Lustre: 25218:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 756.218868] Lustre: 25218:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.221816] Lustre: 25218:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 756.237875] Lustre: 25218:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.247386] Lustre: 25218:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 756.261453] Lustre: 25218:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.961532] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 760.980390] Lustre: 25220:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 760.989463] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 760.993722] Lustre: 25220:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.998868] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 761.002993] Lustre: 25220:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.007106] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 761.011209] Lustre: 25220:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.015077] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 761.018871] Lustre: 25220:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.022583] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 761.026309] Lustre: 25220:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.718083] Lustre: 26925:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 770.731932] Lustre: 26925:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 770.735947] Lustre: 26925:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 770.741818] Lustre: 26925:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 770.750339] Lustre: 26925:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 770.760071] Lustre: 26925:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 770.767361] Lustre: 26925:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 770.773198] Lustre: 26925:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 770.780806] Lustre: 26925:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 770.785833] Lustre: 26925:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 770.794251] Lustre: 26925:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 770.798481] Lustre: 26925:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 819.173358] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 819.182529] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.192075] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 823.776268] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 825.051438] Lustre: server umount lustre-OST07c6 complete [ 834.839575] Lustre: server umount lustre-MDT0000 complete [ 851.248324] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 853.594831] Key type lgssc unregistered [ 853.910727] LNet: 28790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.916336] LNetError: 28790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.939932] LNet: Removed LNI 192.168.201.158@tcp [ 854.799242] Key type .llcrypt unregistered [ 854.802757] Key type ._llcrypt unregistered [ 874.440643] Key type ._llcrypt registered [ 874.443230] Key type .llcrypt registered [ 874.563687] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 875.808939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 875.828713] alg: No test for adler32 (adler32-zlib) [ 876.882331] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 877.149065] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 878.839160] Key type lgssc registered [ 879.992591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 888.965628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 888.985654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.617364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.334311] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 900.076098] LustreError: 30060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.105608] LustreError: 30060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 905.192540] LustreError: 30059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.552702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.104233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 914.963320] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 923.082740] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 929.562647] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 936.132225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 936.557409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.712126] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 951.453899] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 961.082487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 976.355954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.388363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 976.395941] Lustre: Skipped 1 previous similar message [ 981.477463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 981.481670] Lustre: Skipped 1 previous similar message [ 982.019832] Lustre: server umount lustre-OST0000 complete [ 986.595135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.609272] Lustre: Skipped 1 previous similar message [ 986.625392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.201107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 991.213751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.248220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 992.655919] Lustre: server umount lustre-MDT0000 complete [ 996.835307] LustreError: 30059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.847294] LustreError: 30059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 996.872824] LustreError: 30040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441025 with bad export cookie 8314737163925150502 [ 996.874949] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.883565] LustreError: 30040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 997.255730] Lustre: server umount lustre-MDT0001 complete [ 1011.404289] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1014.321270] Key type lgssc unregistered [ 1014.668762] LNet: 33823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.681879] LNetError: 33823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.701895] LNet: Removed LNI 192.168.201.158@tcp [ 1015.513780] Key type .llcrypt unregistered [ 1015.516508] Key type ._llcrypt unregistered [ 1039.358606] Key type ._llcrypt registered [ 1039.364766] Key type .llcrypt registered [ 1039.517545] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1041.721920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1041.762755] alg: No test for adler32 (adler32-zlib) [ 1042.889243] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1043.178756] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1044.879522] Key type lgssc registered [ 1046.004019] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1053.371245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1053.401585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.947042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1059.929662] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1065.447397] LustreError: 35096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.480837] LustreError: 35096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1068.500636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.055159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1073.529972] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1081.560760] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1087.530716] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1093.618881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1094.036375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.003543] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1100.233393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1107.065631] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1120.736924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1120.739036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.754352] LustreError: Skipped 1 previous similar message [ 1120.759348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.769294] Lustre: Skipped 1 previous similar message [ 1123.472892] Lustre: server umount lustre-OST0000 complete [ 1130.465838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1130.468702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.494033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.509758] Lustre: Skipped 1 previous similar message [ 1136.096719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.113221] Lustre: Skipped 2 previous similar messages [ 1140.195638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1140.201087] Lustre: Skipped 1 previous similar message [ 1142.239177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1142.432240] Lustre: server umount lustre-MDT0000 complete [ 1145.318531] LustreError: 35826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1145.339315] LustreError: 35826:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1147.003139] LustreError: 35076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441176 with bad export cookie 1870952526522574954 [ 1147.010169] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.020357] LustreError: 35076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1147.298491] Lustre: server umount lustre-MDT0001 complete [ 1161.932476] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1165.601897] Key type lgssc unregistered [ 1165.942961] LNet: 38637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.950480] LNetError: 38637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.971169] LNet: Removed LNI 192.168.201.158@tcp [ 1167.050315] Key type .llcrypt unregistered [ 1167.051531] Key type ._llcrypt unregistered [ 1195.461653] Key type ._llcrypt registered [ 1195.465895] Key type .llcrypt registered [ 1195.651599] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1196.909540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1197.030164] alg: No test for adler32 (adler32-zlib) [ 1198.468537] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1198.881350] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1200.575815] Key type lgssc registered [ 1202.471786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1212.382735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1212.402887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.884870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.910541] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1223.139030] LustreError: 40057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1223.151147] LustreError: 40057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1228.277596] LustreError: 40057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.534275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.801321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1233.470213] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1241.283551] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1246.823471] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1253.789451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1254.287397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1258.357203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1259.892575] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1268.921672] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1279.970754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1279.982394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1280.006807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1281.934506] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.58@tcp (stopping) [ 1284.067233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1284.075160] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.504171] Lustre: server umount lustre-MDT0000 complete [ 1286.992668] LustreError: 40052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.198654] LustreError: 41851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.232765] LustreError: 41851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1290.368879] LustreError: 40039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441319 with bad export cookie 5213481021067975539 [ 1290.387255] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1290.390400] LustreError: 40039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1292.156733] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (stopping) [ 1292.165370] Lustre: Skipped 2 previous similar messages [ 1294.305562] LustreError: 41851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.310886] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.327102] LustreError: 41851:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1294.370941] Lustre: Skipped 1 previous similar message [ 1296.852454] Lustre: server umount lustre-MDT0001 complete [ 1312.234638] Lustre: server umount lustre-OST0000 complete [ 1329.716332] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1332.698292] Key type lgssc unregistered [ 1333.050705] LNet: 43926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.059603] LNetError: 43926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.083234] LNet: Removed LNI 192.168.201.158@tcp [ 1334.057260] Key type .llcrypt unregistered [ 1334.059571] Key type ._llcrypt unregistered [ 1360.855929] Key type ._llcrypt registered [ 1360.858639] Key type .llcrypt registered [ 1360.939790] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1361.889940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1361.974407] alg: No test for adler32 (adler32-zlib) [ 1362.981330] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1363.196972] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1364.855267] Key type lgssc registered [ 1366.012649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1373.101310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1373.114325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.500056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.037675] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1383.908627] LustreError: 45343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.927927] LustreError: 45343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1388.098787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.499228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1393.081172] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1401.612506] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1408.736443] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1416.679185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1417.050676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.139791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1423.542734] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1432.780612] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1443.809039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1443.822981] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.827602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1443.860228] Lustre: Skipped 1 previous similar message [ 1445.730379] Lustre: lustre-OST0000: Not available for connect from 192.168.201.58@tcp (stopping) [ 1445.737330] Lustre: Skipped 1 previous similar message [ 1447.324839] Lustre: server umount lustre-OST0000 complete [ 1469.408437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.430604] Lustre: Skipped 1 previous similar message [ 1469.432256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1469.453529] Lustre: Skipped 1 previous similar message [ 1473.822818] Lustre: server umount lustre-MDT0000 complete [ 1474.538521] LustreError: 45343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.566704] LustreError: 45343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1478.138564] LustreError: 45323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441507 with bad export cookie 4635881891809561070 [ 1478.139696] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1478.148615] LustreError: 45323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1484.578586] Lustre: server umount lustre-MDT0001 complete [ 1498.668845] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1501.961306] Key type lgssc unregistered [ 1502.376894] LNet: 49055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.402269] LNetError: 49055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.450680] LNet: Removed LNI 192.168.201.158@tcp [ 1503.482680] Key type .llcrypt unregistered [ 1503.484396] Key type ._llcrypt unregistered [ 1529.406654] Key type ._llcrypt registered [ 1529.410915] Key type .llcrypt registered [ 1529.545395] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1530.496923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1530.506182] alg: No test for adler32 (adler32-zlib) [ 1531.583786] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1531.934086] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1533.671182] Key type lgssc registered [ 1534.863663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1543.082621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1543.126331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.924189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1550.406431] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1554.403516] LustreError: 50472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.431191] LustreError: 50472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1559.527229] LustreError: 50473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1559.566169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.913324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1565.059322] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1572.967455] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1579.084084] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1586.974801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1587.423563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.523440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1593.986484] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1602.095839] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1610.839678] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1614.188333] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1626.796332] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 9h20m53s [ 1628.784269] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1631.713968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1631.721166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.739777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.809552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.814523] Lustre: Skipped 4 previous similar messages [ 1634.652178] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.58@tcp (stopping) [ 1637.276059] Lustre: server umount lustre-MDT0000 complete [ 1638.883316] LustreError: 50473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.905244] LustreError: 50473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1641.071216] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1641.809643] LustreError: 50467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1647.945412] LustreError: 50468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1647.985798] LustreError: 50468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1648.472316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.609552] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1648.944752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.017729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1649.971445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1653.678994] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1657.234539] Lustre: DEBUG MARKER: Wait for df (36802) ... [ 1659.041107] Lustre: DEBUG MARKER: done [ 1664.481913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1664.486544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1664.495066] Lustre: Skipped 2 previous similar messages [ 1664.505642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1664.510156] Lustre: Skipped 1 previous similar message [ 1665.512784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.735345] Lustre: server umount lustre-OST0000 complete [ 1675.753151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1675.753800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.767351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.791203] Lustre: Skipped 1 previous similar message [ 1675.805607] Lustre: Skipped 3 previous similar messages [ 1681.379526] Lustre: server umount lustre-MDT0000 complete [ 1682.913116] LustreError: 52425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1682.949967] LustreError: 52425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1686.445032] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441715 with bad export cookie 12689027005775706211 [ 1686.451059] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1686.460760] LustreError: 50454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1686.848996] Lustre: server umount lustre-MDT0001 complete [ 1700.510569] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1704.389450] Key type lgssc unregistered [ 1704.694802] LNet: 55399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.700156] LNetError: 55399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.724641] LNet: Removed LNI 192.168.201.158@tcp [ 1705.577271] Key type .llcrypt unregistered [ 1705.584054] Key type ._llcrypt unregistered [ 1752.729059] Key type ._llcrypt registered [ 1752.738043] Key type .llcrypt registered [ 1752.925385] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1754.729580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1754.749959] alg: No test for adler32 (adler32-zlib) [ 1755.832632] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1756.209186] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1758.000511] Key type lgssc registered [ 1759.461580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1769.082780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1769.125365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.472442] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1770.534476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1770.772303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.434274] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1785.079973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.229053] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1785.279400] Lustre: 57608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1785.614581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1791.440682] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1802.449343] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1810.019753] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1817.649253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1817.951431] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1818.242056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1824.896968] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 1826.472139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1832.798668] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1840.712616] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h24m27s [ 1842.538943] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1850.297243] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1852.864529] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1853.103124] LustreError: 60139:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1873.282744] Lustre: lustre-MDT0000: Client f3b985ce-7d3d-4a6a-970b-a7297062f77b (at 192.168.201.58@tcp) reconnecting [ 1893.135141] LustreError: 60139:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1895.670128] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1898.336836] Lustre: DEBUG MARKER: done [ 1908.195937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1908.211466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.228744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1908.707976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1908.716320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.731323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1910.949630] Lustre: server umount lustre-OST0000 complete [ 1918.953862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1918.961630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.970078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1918.985332] Lustre: Skipped 1 previous similar message [ 1919.011520] Lustre: Skipped 1 previous similar message [ 1921.741645] Lustre: server umount lustre-MDT0000 complete [ 1925.228267] LustreError: 60370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441954 with bad export cookie 8670126854924819855 [ 1925.233945] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1925.235249] LustreError: 60370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1925.624955] Lustre: server umount lustre-MDT0001 complete [ 1940.381195] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 1943.057620] Key type lgssc unregistered [ 1943.479686] LNet: 61953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.489296] LNetError: 61953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.560179] LNet: Removed LNI 192.168.201.158@tcp [ 1945.726406] Key type .llcrypt unregistered [ 1945.728638] Key type ._llcrypt unregistered [ 1991.895781] Key type ._llcrypt registered [ 1991.898309] Key type .llcrypt registered [ 1992.003953] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 1993.303786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.527383] alg: No test for adler32 (adler32-zlib) [ 1994.718766] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1994.971662] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 1996.639217] Key type lgssc registered [ 1998.160628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2001.283102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2001.300339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.509786] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2002.542839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2002.739527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.822803] Lustre: Failing over lustre-MDT0000 [ 2004.220714] Lustre: server umount lustre-MDT0000 complete [ 2012.217846] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:27:18 (1788442038) [ 2020.235044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.810929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.147097] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2035.127539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2035.252902] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2035.299412] Lustre: 64751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2035.615955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2041.749697] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2052.348364] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2059.891579] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2068.956048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2069.317803] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2069.624911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.400429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2076.922179] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2084.760825] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2092.034591] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2102.240981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.260909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2102.262240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2102.264341] Lustre: Skipped 2 previous similar messages [ 2102.286758] Lustre: Skipped 2 previous similar messages [ 2105.857400] Lustre: server umount lustre-MDT0000 complete [ 2110.745785] LustreError: 65486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442139 with bad export cookie 2545566475643956475 [ 2110.756507] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.763456] LustreError: 65486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2111.053821] Lustre: server umount lustre-MDT0001 complete [ 2115.101731] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2121.700520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2128.863141] Lustre: 63675:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442141/real 1788442141] req@00000000001df716 x1875317383801600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442157 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.884454] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.911550] Lustre: 63673:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442144/real 1788442144] req@0000000091f64742 x1875317383801728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2132.924690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.960715] Lustre: 63673:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442146/real 1788442146] req@000000005422946b x1875317383801856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442162 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2135.289498] LustreError: 66599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.303876] LustreError: 66599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2135.385841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.513721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2136.036456] Lustre: 63675:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442149/real 1788442149] req@00000000fced5e2a x1875317383802112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442165 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2137.378102] LustreError: 66600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.429960] LustreError: 66599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.443235] LustreError: 66599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2140.160428] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2141.471861] Lustre: 63674:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442154/real 1788442154] req@00000000849d0399 x1875317383802240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2144.742100] LustreError: 68519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.769529] LustreError: 68519:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2149.532189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.974726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2154.816348] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2155.004527] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2157.100762] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2163.631160] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2170.791422] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2178.548703] Lustre: server umount lustre-MDT0000 complete [ 2180.577572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2180.582255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2180.591223] LustreError: 66600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2182.163537] LustreError: 64744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442211 with bad export cookie 2545566475643958750 [ 2182.184354] LustreError: 64744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2182.185949] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2182.494845] Lustre: server umount lustre-MDT0001 complete [ 2189.471258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2198.752082] Lustre: 63675:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442211/real 1788442211] req@000000000c54ae5d x1875317383829376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2198.779878] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.430712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.007986] LustreError: 63671:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000425a5bb3 x1875317383830656/t0(0) o250->MGC192.168.201.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2207.276062] LustreError: 69514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.350448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2207.391909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2208.365786] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2211.735892] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2222.042343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.690073] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2227.794403] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2235.118188] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2241.581296] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2253.280293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2253.286099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.291298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2253.298330] Lustre: Skipped 2 previous similar messages [ 2253.307440] Lustre: Skipped 2 previous similar messages [ 2254.628210] Lustre: server umount lustre-MDT0000 complete [ 2258.403732] LustreError: 66599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.423923] LustreError: 66599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2258.551184] LustreError: 64744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442287 with bad export cookie 2545566475643960129 [ 2258.556918] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2258.566452] LustreError: 64744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2259.031920] Lustre: server umount lustre-MDT0001 complete [ 2266.415334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2270.518864] Lustre: DEBUG MARKER: files compared the same [ 2282.933382] Lustre: server umount lustre-OST0000 complete [ 2301.084203] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2304.401752] Key type lgssc unregistered [ 2304.773832] LNet: 74483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.781768] LNetError: 74483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.795938] LNet: Removed LNI 192.168.201.158@tcp [ 2305.939452] Key type .llcrypt unregistered [ 2305.946481] Key type ._llcrypt unregistered [ 2333.644852] Key type ._llcrypt registered [ 2333.649732] Key type .llcrypt registered [ 2333.743955] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2335.018629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.142494] alg: No test for adler32 (adler32-zlib) [ 2336.311466] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2336.594551] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2338.343212] Key type lgssc registered [ 2339.378441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2347.763333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2347.777082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.166532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.731898] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug +malloc all [ 2359.777224] LustreError: 75901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.793977] LustreError: 75901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2362.154989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.520510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2367.243711] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug +malloc all [ 2374.800645] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2380.889972] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2387.352402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2387.691359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.732748] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug +malloc all [ 2394.856292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2401.331657] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2415.584064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.592108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.619881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2417.634445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2417.637304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.157393] Lustre: server umount lustre-OST0000 complete [ 2427.872298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.885252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2427.887767] Lustre: Skipped 1 previous similar message [ 2432.038838] Lustre: server umount lustre-MDT0000 complete [ 2432.996419] LustreError: 75897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.028090] LustreError: 75897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2436.576878] LustreError: 75883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442465 with bad export cookie 1044463764562444108 [ 2436.582743] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2436.586758] LustreError: 75883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2437.110895] Lustre: server umount lustre-MDT0001 complete [ 2447.836067] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2450.241666] Key type lgssc unregistered [ 2450.506527] LNet: 79456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.511527] LNetError: 79456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2450.526396] LNet: Removed LNI 192.168.201.158@tcp [ 2451.317855] Key type .llcrypt unregistered [ 2451.320046] Key type ._llcrypt unregistered [ 2474.618345] Key type ._llcrypt registered [ 2474.620418] Key type .llcrypt registered [ 2474.699072] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2475.737119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2475.896557] alg: No test for adler32 (adler32-zlib) [ 2476.890664] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2477.108394] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2478.744408] Key type lgssc registered [ 2479.557257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2486.774830] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2486.796180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2502.369983] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2507.491763] LustreError: 80848:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [ 2534.111426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2540.077349] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2548.574944] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2555.489582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.584757] Lustre: *** cfs_fail_loc=706, val=0*** [ 2555.609899] LustreError: 80183:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000837986f2 x1875317889173760/t0(0) o250->MGC192.168.201.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2555.944678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2556.012673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2560.063590] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2560.994817] LustreError: 80874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.017571] LustreError: 80874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2566.117545] LustreError: 80873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.192681] LustreError: 80874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.218536] LustreError: 80874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2568.921381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2569.511552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.718216] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2582.071049] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2588.001466] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2590.178121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.185970] Lustre: Skipped 1 previous similar message [ 2590.190843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.192354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2590.199472] Lustre: Skipped 1 previous similar message [ 2595.301664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2595.307538] Lustre: Skipped 1 previous similar message [ 2596.012581] Lustre: server umount lustre-OST0000 complete [ 2600.417626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2600.418284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.431815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2600.449546] Lustre: Skipped 1 previous similar message [ 2605.540155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2605.545325] Lustre: Skipped 2 previous similar messages [ 2605.988963] Lustre: server umount lustre-MDT0000 complete [ 2609.237027] LustreError: 80854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442638 with bad export cookie 8872729636443402186 [ 2609.238916] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.246735] LustreError: 80854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2609.558350] Lustre: server umount lustre-MDT0001 complete [ 2620.167177] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 2623.008164] Key type lgssc unregistered [ 2623.301451] LNet: 84465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2623.309750] LNetError: 84465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2623.352813] LNet: Removed LNI 192.168.201.158@tcp [ 2624.151963] Key type .llcrypt unregistered [ 2624.154081] Key type ._llcrypt unregistered [ 2655.933344] Key type ._llcrypt registered [ 2655.936364] Key type .llcrypt registered [ 2656.202792] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2658.279232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2658.445509] alg: No test for adler32 (adler32-zlib) [ 2659.846319] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2660.652798] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 2662.679404] Key type lgssc registered [ 2664.857151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2673.719329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2673.737557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.784723] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2685.910754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.548722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2691.072375] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2691.552139] LustreError: 86571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.572575] LustreError: 86571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2696.700251] LustreError: 86569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.716660] LustreError: 86569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2699.265271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2699.474697] LustreError: 86571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.498112] LustreError: 86571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2699.511966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.867377] LustreError: 86569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.879906] LustreError: 86569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2706.001505] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2709.990785] LustreError: 86574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.014637] LustreError: 86574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2714.793738] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2714.803267] Lustre: MGS: Client 7e2b8510-ea35-4d22-8028-ba3b30ba7b46 (at 0@lo) reconnecting [ 2714.820860] LustreError: 86585:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2714.992855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.089126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2719.372095] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2735.590361] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.601595] Lustre: Skipped 1 previous similar message [ 2735.604707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2735.609320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2738.329738] Lustre: server umount lustre-OST0000 complete [ 2745.828200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2745.830957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2745.852336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2745.856904] Lustre: Skipped 2 previous similar messages [ 2749.350330] Lustre: server umount lustre-MDT0000 complete [ 2750.946233] LustreError: 87318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.974431] LustreError: 87318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2755.920809] Lustre: server umount lustre-MDT0001 complete [ 2762.111324] Lustre: server umount MGS complete [ 2778.964178] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:40:05 (1788442805) [ 2809.251519] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 2820.756633] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 2858.145650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.957650] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2873.105211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.294830] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2873.330767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2873.396238] Lustre: lustre-MDT0001: new disk, initializing [ 2873.477816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2873.495848] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2873.504794] Lustre: 93746:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2873.513353] LustreError: 93746:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2877.895344] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2888.845389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.084319] Lustre: lustre-OST0000: new disk, initializing [ 2889.086933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2889.095986] Lustre: 94511:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2889.161165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.873689] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2897.504633] Lustre: 94952:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000e38458f5 x1875318081101824/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788442942 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2904.593804] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2911.673894] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2911.687849] Lustre: MGS: Client f7f4fc9c-5942-4c14-b3c4-7d6131afabce (at 0@lo) reconnecting [ 2911.726150] LustreError: 93732:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2911.768430] Lustre: 93011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2911.810591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2911.913882] Lustre: lustre-MDT0000: new disk, initializing [ 2912.055818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.075243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2912.098800] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2913.121365] Lustre: 95580:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000613a02dc x1875318081108608/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788442958 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2916.641443] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 2918.306691] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2922.222612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2922.228531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2922.374619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2948.069945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2948.075450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.083542] Lustre: Skipped 1 previous similar message [ 2948.091558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.625068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2950.636744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.650860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2957.282778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2957.297781] Lustre: Skipped 3 previous similar messages [ 2962.402546] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2962.855848] Lustre: server umount lustre-OST0000 complete [ 2972.642991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2972.682475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2972.691108] Lustre: Skipped 3 previous similar messages [ 2975.814515] Lustre: server umount lustre-MDT0000 complete [ 2977.761982] LustreError: 93717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.777724] LustreError: 93717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2987.601465] Lustre: server umount lustre-MDT0001 complete [ 2993.047788] Lustre: server umount MGS complete [ 3009.819738] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:43:56 (1788443036) [ 3030.638420] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3033.938210] Key type lgssc unregistered [ 3034.321559] LNet: 98087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3034.334204] LNetError: 98087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3034.361203] LNet: Removed LNI 192.168.201.158@tcp [ 3035.480766] Key type .llcrypt unregistered [ 3035.483175] Key type ._llcrypt unregistered [ 3055.816783] Key type ._llcrypt registered [ 3055.818965] Key type .llcrypt registered [ 3056.014928] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3056.964510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3057.155601] alg: No test for adler32 (adler32-zlib) [ 3058.293439] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3058.657364] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3060.327272] Key type lgssc registered [ 3061.554505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3069.714773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3069.736721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3071.287628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.751607] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3081.698517] LustreError: 99310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.723386] LustreError: 99310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3085.025071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.412250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3090.127442] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3097.583728] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3104.595992] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3112.453745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3112.886640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.978644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3120.252305] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3129.239382] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3140.580524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3140.592619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.600680] Lustre: Skipped 1 previous similar message [ 3140.609880] Lustre: Skipped 1 previous similar message [ 3145.710911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3145.721302] Lustre: Skipped 1 previous similar message [ 3146.195530] Lustre: server umount lustre-OST0000 complete [ 3151.842170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3151.876794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.895942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3155.937517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3155.962988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.946853] Lustre: server umount lustre-MDT0000 complete [ 3161.063491] LustreError: 99305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.074069] LustreError: 99305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3161.904841] LustreError: 99293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443191 with bad export cookie 5518034559768480043 [ 3161.909553] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3161.917477] LustreError: 99293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3162.304570] Lustre: server umount lustre-MDT0001 complete [ 3175.022753] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3178.805600] Key type lgssc unregistered [ 3179.286542] LNet: 102851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3179.289129] LNetError: 102851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3179.327839] LNet: Removed LNI 192.168.201.158@tcp [ 3179.827558] Key type .llcrypt unregistered [ 3179.830760] Key type ._llcrypt unregistered [ 3199.812785] Key type ._llcrypt registered [ 3199.815922] Key type .llcrypt registered [ 3199.928449] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3201.709493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3201.762458] alg: No test for adler32 (adler32-zlib) [ 3202.986959] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3203.408633] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3205.135179] Key type lgssc registered [ 3206.466273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3213.039151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3213.048127] LustreError: 103930:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3213.110636] LustreError: 103940:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3213.116113] LustreError: 103930:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4937 [ 3213.127390] LustreError: 103930:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3213.132336] LustreError: 103930:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3213.146114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.656170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3223.810238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.031949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3224.160164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3228.550396] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3234.441995] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing lsmod [ 3236.912874] LustreError: 105143:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3236.927833] LustreError: 105145:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3236.933989] LustreError: 105143:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3236.940100] LustreError: 105143:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3236.942735] LustreError: 105143:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3236.959818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.338845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.407900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3244.513078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3244.514031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.522463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3244.531724] Lustre: Skipped 2 previous similar messages [ 3244.548381] Lustre: Skipped 2 previous similar messages [ 3246.417717] Lustre: server umount lustre-MDT0000 complete [ 3249.636516] LustreError: 103978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3249.664524] LustreError: 103978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3250.080401] LustreError: 103957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443279 with bad export cookie 6539391533465024780 [ 3250.094322] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3251.807729] LustreError: 104408:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a3ed6fe6 x1875318650458112/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3251.962831] Lustre: server umount lustre-MDT0001 complete [ 3266.858467] Lustre: server umount lustre-OST0000 complete [ 3274.964472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.302615] LustreError: 106210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.390883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3279.767639] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3280.872680] LustreError: 106211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.969944] LustreError: 106210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.932158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.263408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3293.145474] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3301.362891] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3308.235046] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3315.016381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3315.351902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.524841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3322.549516] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3329.698953] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3342.818096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3342.820648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.829128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3344.353466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3349.232724] Lustre: server umount lustre-OST0000 complete [ 3355.105361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3355.118592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3355.137279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3355.140169] Lustre: Skipped 2 previous similar messages [ 3359.725128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.726227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.775875] Lustre: Skipped 1 previous similar message [ 3361.452166] Lustre: server umount lustre-MDT0000 complete [ 3364.832941] LustreError: 106211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.875867] LustreError: 106211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3366.385808] LustreError: 106191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443395 with bad export cookie 6539391533465025711 [ 3366.390207] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3366.412417] LustreError: 106191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3367.237994] Lustre: server umount lustre-MDT0001 complete [ 3384.433905] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3388.871207] Key type lgssc unregistered [ 3389.430926] LNet: 109790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3389.452494] LNetError: 109790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3390.527099] LNet: Removed LNI 192.168.201.158@tcp [ 3392.132970] Key type .llcrypt unregistered [ 3392.138989] Key type ._llcrypt unregistered [ 3425.141611] Key type ._llcrypt registered [ 3425.143071] Key type .llcrypt registered [ 3425.260463] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3426.449640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3426.555755] alg: No test for adler32 (adler32-zlib) [ 3427.767275] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3428.032380] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3429.711255] Key type lgssc registered [ 3430.891558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3438.671504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3438.685301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.171103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.284487] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3450.337169] LustreError: 111210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3450.356296] LustreError: 111210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3455.316027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.460234] LustreError: 111211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3455.954118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3460.849416] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3469.334240] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3477.142419] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3484.567717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3484.823334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3485.906566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3492.643777] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3500.852775] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3506.156314] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3510.451166] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3515.880092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3515.881546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.885061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3515.924411] Lustre: Skipped 1 previous similar message [ 3520.996786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3521.022505] Lustre: Skipped 2 previous similar messages [ 3522.093694] Lustre: server umount lustre-OST0000 complete [ 3527.659885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3527.676715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3527.701545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.234418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.236062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3532.652838] Lustre: server umount lustre-MDT0000 complete [ 3536.353827] LustreError: 111211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.383249] LustreError: 111211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3536.612460] LustreError: 113744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443565 with bad export cookie 16348435878873070351 [ 3536.614523] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3536.628710] LustreError: 113744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3536.975376] Lustre: server umount lustre-MDT0001 complete [ 3548.107705] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3552.223972] Key type lgssc unregistered [ 3552.703792] LNet: 114854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3552.715112] LNetError: 114854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3552.748288] LNet: Removed LNI 192.168.201.158@tcp [ 3553.790761] Key type .llcrypt unregistered [ 3553.793387] Key type ._llcrypt unregistered [ 3577.433713] Key type ._llcrypt registered [ 3577.435780] Key type .llcrypt registered [ 3577.561897] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3579.149161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3579.528786] alg: No test for adler32 (adler32-zlib) [ 3580.519822] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3580.785405] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3582.663711] Key type lgssc registered [ 3583.899834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3593.491685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3593.508800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.997418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3599.437915] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3605.474410] LustreError: 116084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3605.487368] LustreError: 116084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3608.627623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.183760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3614.499570] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3623.370687] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3629.587205] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3637.230476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3637.583575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3640.715381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3644.716961] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3653.057783] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3662.307257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3662.308812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3662.312807] Lustre: Skipped 1 previous similar message [ 3662.321819] Lustre: Skipped 1 previous similar message [ 3667.437981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3667.451166] Lustre: Skipped 1 previous similar message [ 3668.165333] Lustre: server umount lustre-OST0000 complete [ 3675.618892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3675.647162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.679843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3677.680151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.686748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3677.721505] Lustre: Skipped 1 previous similar message [ 3682.786813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3687.391354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3687.585455] Lustre: server umount lustre-MDT0000 complete [ 3687.910821] LustreError: 116085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.944486] LustreError: 116085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3691.955492] LustreError: 116066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443721 with bad export cookie 11860857695190865588 [ 3691.957795] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3691.966878] LustreError: 116066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3692.636454] Lustre: server umount lustre-MDT0001 complete [ 3706.623324] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3709.695046] Key type lgssc unregistered [ 3709.920908] LNet: 119624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3709.930726] LNetError: 119624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3709.939957] LNet: Removed LNI 192.168.201.158@tcp [ 3710.728146] Key type .llcrypt unregistered [ 3710.740105] Key type ._llcrypt unregistered [ 3736.731868] Key type ._llcrypt registered [ 3736.734366] Key type .llcrypt registered [ 3736.815616] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3738.522529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3738.551160] alg: No test for adler32 (adler32-zlib) [ 3739.550341] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3739.755865] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3741.423143] Key type lgssc registered [ 3742.725440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3750.065652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3750.080863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.483922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.169819] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3761.634215] LustreError: 121043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3761.651427] LustreError: 121043:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3763.942259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3764.340368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3768.898711] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3776.337692] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3782.200056] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3789.264103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3789.627295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.531925] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3796.791272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3803.117862] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3812.874312] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3816.018135] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3816.022812] Lustre: Skipped 2 previous similar messages [ 3819.597951] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3819.632589] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3819.927628] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3819.935363] Lustre: Skipped 2 previous similar messages [ 3826.847148] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3826.854078] Lustre: Skipped 1 previous similar message [ 3832.545958] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.58@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3832.555449] Lustre: Skipped 2 previous similar messages [ 3837.054697] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.58@tcp 192.168.0.[2,10]@tcp [ 3840.446529] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.58/24@tcp in log lustre-MDT0000 [ 3840.454197] Lustre: Skipped 2 previous similar messages [ 3844.453867] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.58/24@tcp [ 3844.460238] Lustre: Skipped 1 previous similar message [ 3850.214314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.222069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.233839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3850.256479] Lustre: Skipped 1 previous similar message [ 3855.331801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3855.342418] Lustre: Skipped 2 previous similar messages [ 3855.552712] Lustre: server umount lustre-OST0000 complete [ 3860.450127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.468338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3860.482881] Lustre: Skipped 1 previous similar message [ 3860.501718] Lustre: Skipped 1 previous similar message [ 3865.409566] Lustre: server umount lustre-MDT0000 complete [ 3865.577674] LustreError: 121042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.596913] LustreError: 121042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3869.135148] LustreError: 121023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443898 with bad export cookie 15103975482310808039 [ 3869.139494] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3869.147903] LustreError: 121023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3869.440806] Lustre: server umount lustre-MDT0001 complete [ 3881.164631] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 3884.094480] Key type lgssc unregistered [ 3884.307685] LNet: 125943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3884.311750] LNetError: 125943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3884.320871] LNet: Removed LNI 192.168.201.158@tcp [ 3885.025730] Key type .llcrypt unregistered [ 3885.027696] Key type ._llcrypt unregistered [ 3914.314583] Key type ._llcrypt registered [ 3914.316695] Key type .llcrypt registered [ 3914.424934] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 3915.541065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3915.568801] alg: No test for adler32 (adler32-zlib) [ 3916.848676] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3917.074552] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 3918.799957] Key type lgssc registered [ 3921.066437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3932.764700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3932.790385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.630288] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3934.701944] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.58/24@tcp [ 3934.705152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3941.677157] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3944.939491] LustreError: 127505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.960204] LustreError: 127505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3950.050534] LustreError: 127504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.026926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.335599] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3952.401610] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.58/24@tcp [ 3952.491717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3958.867956] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 3969.818446] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3977.504770] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3986.018922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3986.289119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3993.997631] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4002.549406] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4005.750563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4018.144966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4018.159818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.170782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4018.195671] Lustre: Skipped 1 previous similar message [ 4018.230375] Lustre: Skipped 1 previous similar message [ 4021.219282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4021.232979] Lustre: Skipped 1 previous similar message [ 4023.745469] Lustre: server umount lustre-OST0000 complete [ 4029.417113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4029.446395] LustreError: Skipped 1 previous similar message [ 4029.461636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.490061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4031.484637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4031.507793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4031.520058] Lustre: Skipped 1 previous similar message [ 4035.964974] Lustre: server umount lustre-MDT0000 complete [ 4036.599724] LustreError: 127504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4036.648451] LustreError: 127504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4041.699560] LustreError: 128609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4041.714700] LustreError: 128609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4041.893087] LustreError: 130143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444070 with bad export cookie 14759256042282237267 [ 4041.894619] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4041.899563] LustreError: 130143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4042.248537] Lustre: server umount lustre-MDT0001 complete [ 4054.370990] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4059.668440] Key type lgssc unregistered [ 4059.936764] LNet: 131113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4059.947863] LNetError: 131113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4060.965465] LNet: Removed LNI 192.168.201.158@tcp [ 4062.510310] Key type .llcrypt unregistered [ 4062.513975] Key type ._llcrypt unregistered [ 4094.576346] Key type ._llcrypt registered [ 4094.579557] Key type .llcrypt registered [ 4094.820671] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 4114.146701] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4116.007378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4116.065804] alg: No test for adler32 (adler32-zlib) [ 4117.167466] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4117.421773] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4119.079664] Key type lgssc registered [ 4120.690757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4159.395354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4159.429163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.703300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4160.740656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4160.814499] Lustre: lustre-MDT0000: new disk, initializing [ 4160.928919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.976727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4166.397190] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4177.818118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4177.982817] Lustre: 134532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4178.046359] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4178.059782] Lustre: Skipped 1 previous similar message [ 4178.142326] Lustre: lustre-MDT0001: new disk, initializing [ 4178.276981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4178.318306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4178.339114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4184.180953] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4195.501556] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4201.485287] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4208.860881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4209.010223] Lustre: lustre-OST0000: new disk, initializing [ 4209.012344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4209.024374] Lustre: 136499:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4209.109776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.360570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4215.366370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4215.501712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4215.887453] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4226.396536] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4231.767512] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4232.015591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4236.243986] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4236.529071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4241.381968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4241.383631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.390317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4241.426215] Lustre: Skipped 1 previous similar message [ 4245.619066] Lustre: server umount lustre-OST0000 complete [ 4250.083725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4250.091694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.133280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4250.147779] Lustre: Skipped 1 previous similar message [ 4251.616884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4251.618644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.401225] Lustre: server umount lustre-MDT0000 complete [ 4260.422116] LustreError: 134524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444289 with bad export cookie 12159587209729951334 [ 4260.427962] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4260.436396] LustreError: 134524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4261.215376] Lustre: server umount lustre-MDT0001 complete [ 4273.145203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.993791] LustreError: 138720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.018858] LustreError: 138720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4274.086314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4279.274927] LustreError: 138720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.290351] LustreError: 138720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4281.183812] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4284.384509] LustreError: 138721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4289.505503] LustreError: 138720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.910791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.287500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4297.731137] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4305.762406] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4312.789291] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4319.421773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4319.803770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.856704] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4334.856465] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4340.638789] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4345.358468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4350.313190] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4350.660872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4359.398457] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4359.576274] Lustre: lustre-OST0001: new disk, initializing [ 4359.580252] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4359.592789] Lustre: 142121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4359.707133] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4366.368928] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4370.100979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4370.116078] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4370.234327] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4375.070726] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4375.343472] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4380.485478] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4380.756630] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4386.363907] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97ba10916000.ost_server_uuid 50 [ 4388.191520] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97ba10916000.ost_server_uuid in IDLE state after 0 sec [ 4399.303128] Lustre: server umount lustre-OST0001 complete [ 4401.121756] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4401.123680] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.148324] LustreError: Skipped 1 previous similar message [ 4401.159952] LustreError: 141470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4401.162994] Lustre: Skipped 1 previous similar message [ 4401.180307] LustreError: 141470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4405.730066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4405.733578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4405.777107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4405.787455] Lustre: Skipped 1 previous similar message [ 4410.163523] Lustre: server umount lustre-OST0000 complete [ 4415.457472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4415.469980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4415.499118] Lustre: Skipped 1 previous similar message [ 4415.504701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4415.518476] Lustre: Skipped 2 previous similar messages [ 4421.131372] Lustre: server umount lustre-MDT0000 complete [ 4421.600986] LustreError: 138721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4421.647402] LustreError: 138721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4425.791509] LustreError: 142120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444454 with bad export cookie 12159587209729952216 [ 4425.805989] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4425.809946] LustreError: 142120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4426.175950] Lustre: server umount lustre-MDT0001 complete [ 4448.488633] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4451.218098] Key type lgssc unregistered [ 4451.531704] LNet: 145127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4451.548844] LNetError: 145127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4451.570724] LNet: Removed LNI 192.168.201.158@tcp [ 4452.450749] Key type .llcrypt unregistered [ 4452.453360] Key type ._llcrypt unregistered [ 4494.574696] Key type ._llcrypt registered [ 4494.577132] Key type .llcrypt registered [ 4494.707871] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 4512.214973] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4513.821867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4513.842629] alg: No test for adler32 (adler32-zlib) [ 4515.233205] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4515.600972] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4517.343145] Key type lgssc registered [ 4519.060438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4556.647779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4556.668727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.902258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4557.928874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4558.014835] Lustre: lustre-MDT0000: new disk, initializing [ 4558.083540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.096900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4562.354886] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4574.048415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4574.195888] Lustre: 149285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4574.247266] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4574.251144] Lustre: Skipped 1 previous similar message [ 4574.370938] Lustre: lustre-MDT0001: new disk, initializing [ 4574.464081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4574.535256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4574.550870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4580.113773] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4592.132348] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4599.153478] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4606.198871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4606.419903] Lustre: lustre-OST0000: new disk, initializing [ 4606.423819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4606.432653] Lustre: 151251:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4606.530456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4613.586344] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4614.222610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4614.231721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4614.364820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4622.829641] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4628.050641] Lustre: Failing over lustre-OST0000 [ 4628.284864] Lustre: server umount lustre-OST0000 complete [ 4646.239149] Lustre: 147506:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444659/real 1788444659] req@0000000039a41666 x1875320026204416/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788444675 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4646.270631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.336104] Lustre: 147508:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444659/real 1788444659] req@000000001dd9058b x1875320026204288/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788444675 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4647.366557] Lustre: 147508:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4648.470374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4648.716917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4648.745564] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4651.423242] Lustre: 147505:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444664/real 1788444664] req@000000001327a930 x1875320026205440/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788444680 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4656.848236] Lustre: 147505:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444669/real 1788444669] req@00000000eb9625c6 x1875320026206592/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788444685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4656.876402] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4656.879143] Lustre: 147505:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4656.931382] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4657.533657] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4657.538912] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4661.219848] Lustre: Failing over lustre-MDT0000 [ 4661.486835] Lustre: server umount lustre-MDT0000 complete [ 4661.735774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4661.743691] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4661.755237] Lustre: Skipped 1 previous similar message [ 4662.241739] Lustre: 147508:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444674/real 1788444674] req@00000000c3c13ee6 x1875320026207744/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788444690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4667.361043] LustreError: 149297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4667.387245] LustreError: 149297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4672.480788] LustreError: 149296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4672.503131] LustreError: 149296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4677.602697] LustreError: 149297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4677.614524] LustreError: 149297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4678.623165] Lustre: 147506:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444691/real 1788444691] req@00000000a41f03db x1875320026217600/t0(0) o400->MGC192.168.201.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788444707 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4678.666963] Lustre: 147506:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4678.683239] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4682.154975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.081598] LustreError: 151476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4683.111093] LustreError: 151476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4688.255656] LustreError: 151476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4688.263899] LustreError: 151476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4688.893490] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x5233aebc65481de3 [ 4688.959031] Lustre: MGC192.168.201.158@tcp: Connection restored to 0@lo (at 0@lo) [ 4688.976782] Lustre: Skipped 1 previous similar message [ 4689.176708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.229385] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4693.331112] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4694.513059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4694.563523] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4694.663781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4696.489387] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4708.866263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4708.869773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4708.877536] Lustre: Skipped 2 previous similar messages [ 4708.880910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4709.862906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4714.647903] Lustre: server umount lustre-OST0000 complete [ 4719.995974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.054095] Lustre: Skipped 1 previous similar message [ 4720.070171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4720.073041] Lustre: Skipped 1 previous similar message [ 4725.227997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4725.247629] Lustre: Skipped 1 previous similar message [ 4726.440391] Lustre: server umount lustre-MDT0000 complete [ 4730.279454] LustreError: 149275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444759 with bad export cookie 5923270059098643939 [ 4730.280448] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4730.290414] LustreError: 149275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4730.508556] Lustre: server umount lustre-MDT0001 complete [ 4743.558588] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 4747.081364] Key type lgssc unregistered [ 4747.389343] LNet: 154913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4747.403034] LNetError: 154913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4747.421504] LNet: Removed LNI 192.168.201.158@tcp [ 4748.448498] Key type .llcrypt unregistered [ 4748.453880] Key type ._llcrypt unregistered [ 4789.940615] Key type ._llcrypt registered [ 4789.949650] Key type .llcrypt registered [ 4790.122379] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4792.256726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4792.503282] alg: No test for adler32 (adler32-zlib) [ 4793.756609] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4793.980447] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 4795.719278] Key type lgssc registered [ 4797.141233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4804.678892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4804.702191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.288574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4810.973556] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4816.873813] LustreError: 156434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4816.902470] LustreError: 156434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4821.119521] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4821.987355] LustreError: 156435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4826.141232] LustreError: 156463:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4826.162814] LustreError: 156463:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4826.173496] LustreError: 156463:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4826.626874] Lustre: server umount lustre-MDT0000 complete [ 4844.814494] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 4856.558759] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 4899.989739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.260949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4900.288148] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4900.355903] Lustre: lustre-MDT0000: new disk, initializing [ 4900.431145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.456880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4905.710312] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4916.500407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4916.585853] Lustre: 159771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4916.635311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4916.638751] Lustre: Skipped 1 previous similar message [ 4916.739249] Lustre: lustre-MDT0001: new disk, initializing [ 4916.853730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4916.893563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4916.909975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4921.665139] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4932.965873] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4940.541166] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4949.119322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4949.356384] Lustre: lustre-OST0000: new disk, initializing [ 4949.359144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4949.363893] Lustre: 161737:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4949.469492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4956.259168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4956.282091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4956.499982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4956.542161] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 4967.570598] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4972.599885] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4974.611108] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:16:41 (1788445001) [ 5238.240073] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5238.250248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5238.257813] Lustre: Skipped 1 previous similar message [ 5238.271995] Lustre: Skipped 2 previous similar messages [ 5240.924347] Lustre: server umount lustre-MDT0000 complete [ 5243.362950] LustreError: 159775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5243.393437] LustreError: 159775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5245.065781] LustreError: 159764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445274 with bad export cookie 827831400945742897 [ 5245.068589] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5245.075831] LustreError: 159764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5248.480266] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5248.481970] LustreError: 159777:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5248.511578] Lustre: Skipped 1 previous similar message [ 5248.512063] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5250.529135] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5251.374103] Lustre: server umount lustre-MDT0001 complete [ 5262.052172] Lustre: server umount lustre-OST0000 complete [ 5276.929188] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 5289.102707] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5327.650790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.018235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5328.080648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5328.220451] Lustre: lustre-MDT0000: new disk, initializing [ 5328.367773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5328.431900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5335.827852] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5347.629384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.807169] Lustre: 166403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5347.870727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5347.878424] Lustre: Skipped 1 previous similar message [ 5348.033025] Lustre: lustre-MDT0001: new disk, initializing [ 5348.242784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5348.340091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5348.353577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5354.317983] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5367.726730] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5375.617044] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5385.073171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5385.502511] Lustre: lustre-OST0000: new disk, initializing [ 5385.511666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5385.522702] Lustre: 168373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5385.665218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5391.136352] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5395.538123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5395.550849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5395.678822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5401.019274] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5407.153836] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5407.316540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5412.832935] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5413.071151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5416.416094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5416.422552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5416.432491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5421.520646] Lustre: server umount lustre-OST0000 complete [ 5426.619237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.636542] Lustre: Skipped 1 previous similar message [ 5426.651265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5426.657128] Lustre: Skipped 1 previous similar message [ 5433.048171] Lustre: server umount lustre-MDT0000 complete [ 5436.908706] LustreError: 167774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5436.930488] LustreError: 167774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5437.206466] LustreError: 166398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445466 with bad export cookie 827831400945901727 [ 5437.211585] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5437.216638] LustreError: 166398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5437.594351] Lustre: server umount lustre-MDT0001 complete [ 5447.159289] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:24:34 (1788445474) [ 5464.506713] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 5475.265822] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5510.739915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5511.017316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5511.041278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5511.100812] Lustre: lustre-MDT0000: new disk, initializing [ 5511.165306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.182959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5516.037364] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5526.660731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.841370] Lustre: 173130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5527.330394] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5532.698157] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5545.595628] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5555.061242] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5569.349157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5569.916962] Lustre: lustre-OST0000: new disk, initializing [ 5569.923411] Lustre: Skipped 1 previous similar message [ 5569.927794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5569.932388] Lustre: Skipped 2 previous similar messages [ 5569.936955] Lustre: 175108:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5578.816768] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5579.332800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5579.362315] Lustre: Skipped 1 previous similar message [ 5579.373056] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5579.575603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5588.975941] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5600.240621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5600.249552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.256460] LustreError: Skipped 2 previous similar messages [ 5600.261701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5600.274515] Lustre: Skipped 2 previous similar messages [ 5600.302956] Lustre: Skipped 3 previous similar messages [ 5603.246719] Lustre: server umount lustre-OST0000 complete [ 5609.440753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5609.445049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5613.859586] Lustre: server umount lustre-MDT0000 complete [ 5615.591672] LustreError: 174016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.629888] LustreError: 174016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5617.950423] LustreError: 173121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445647 with bad export cookie 827831400945902609 [ 5617.958027] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5617.965628] LustreError: 173121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5618.335277] Lustre: server umount lustre-MDT0001 complete [ 5627.981780] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:27:35 (1788445655) [ 5644.660162] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 5653.819448] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5697.483898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.828141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5697.857431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5697.955218] Lustre: lustre-MDT0000: new disk, initializing [ 5698.032474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.036096] Lustre: Skipped 2 previous similar messages [ 5698.052536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5703.168662] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5715.405431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5715.568251] Lustre: 179432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5715.875720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5720.140384] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5730.285404] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5736.566113] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5743.426611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5743.830047] Lustre: 181408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5750.477247] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5750.846332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5750.944983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5760.526700] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5771.748194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5771.748458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.764967] LustreError: Skipped 1 previous similar message [ 5771.770454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.785680] Lustre: Skipped 2 previous similar messages [ 5771.795952] Lustre: Skipped 4 previous similar messages [ 5774.455734] Lustre: server umount lustre-OST0000 complete [ 5792.223697] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5792.539435] Lustre: server umount lustre-MDT0000 complete [ 5796.336498] LustreError: 179441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5796.359351] LustreError: 179441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5797.126904] LustreError: 179426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445826 with bad export cookie 827831400945903687 [ 5797.134248] LustreError: 179426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5797.134957] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5797.740150] Lustre: server umount lustre-MDT0001 complete [ 5810.269525] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 5814.025849] Key type lgssc unregistered [ 5814.488428] LNet: 183269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5814.504076] LNetError: 183269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5814.519657] LNet: Removed LNI 192.168.201.158@tcp [ 5815.529581] Key type .llcrypt unregistered [ 5815.531621] Key type ._llcrypt unregistered [ 5841.624788] Key type ._llcrypt registered [ 5841.630804] Key type .llcrypt registered [ 5841.827789] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 5843.364750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5843.521768] alg: No test for adler32 (adler32-zlib) [ 5844.687587] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5844.969397] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 5846.687228] Key type lgssc registered [ 5847.902665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5855.904852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5855.940851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5857.492493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5862.172915] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5866.992512] LustreError: 184687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.016034] LustreError: 184687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5871.254494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5871.725846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5877.123580] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5885.856871] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5892.537653] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5899.209690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5899.520215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.607228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5905.533464] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 5913.742222] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5936.611036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.616666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5936.626778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.639214] Lustre: Skipped 1 previous similar message [ 5940.248850] Lustre: server umount lustre-OST0000 complete [ 5946.849445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.858474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5946.877153] Lustre: Skipped 2 previous similar messages [ 5946.891268] Lustre: Skipped 1 previous similar message [ 5951.969384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.975853] Lustre: Skipped 1 previous similar message [ 5956.065148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5956.073352] Lustre: Skipped 1 previous similar message [ 5958.113968] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5958.337142] Lustre: server umount lustre-MDT0000 complete [ 5961.186892] LustreError: 184682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.199167] LustreError: 184682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5962.037726] LustreError: 185425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445991 with bad export cookie 14663025106750998950 [ 5962.045737] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5962.047063] LustreError: 185425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5962.280298] Lustre: server umount lustre-MDT0001 complete [ 5980.497864] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 5983.757254] Key type lgssc unregistered [ 5984.261755] LNet: 188271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5984.266054] LNetError: 188271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5984.286769] LNet: Removed LNI 192.168.201.158@tcp [ 5985.272371] Key type .llcrypt unregistered [ 5985.276911] Key type ._llcrypt unregistered [ 6013.971574] Key type ._llcrypt registered [ 6013.975503] Key type .llcrypt registered [ 6014.094161] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 6015.719054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6015.763211] alg: No test for adler32 (adler32-zlib) [ 6017.003341] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6017.305744] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 6018.999154] Key type lgssc registered [ 6020.500509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6030.164363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6030.179429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6031.610847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.624423] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6041.123478] LustreError: 189692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6041.151216] LustreError: 189692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6046.177409] LustreError: 189691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6049.797325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.280804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6055.923806] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6064.609312] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6071.002522] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6077.495685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6077.844902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6084.182458] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6091.295166] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6101.392345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6107.482655] Lustre: lustre-OST0000: Not available for connect from 192.168.201.58@tcp (stopping) [ 6110.688326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6110.696771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.718831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.715964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6111.718746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.721660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6113.164567] Lustre: server umount lustre-OST0000 complete [ 6121.216968] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff97ba18f1b800.ost_server_uuid 50 [ 6123.093977] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6141.413240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.424070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6141.425954] Lustre: Skipped 2 previous similar messages [ 6144.508880] Lustre: server umount lustre-MDT0000 complete [ 6146.528283] LustreError: 189691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6146.547643] LustreError: 189691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6147.950945] LustreError: 189672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446177 with bad export cookie 15892280700255266675 [ 6147.952369] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6147.963538] LustreError: 189672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6148.486324] Lustre: server umount lustre-MDT0001 complete [ 6157.487967] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:36:24 (1788446184) [ 6165.397730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6165.892420] LustreError: 193631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.029123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6171.117065] LustreError: 193632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.969249] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6181.351625] LustreError: 193632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6181.373152] LustreError: 193632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6182.067133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.582540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6187.359250] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6194.544185] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6200.570588] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6207.384290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6207.688300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.828125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6213.554816] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6220.894463] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6227.725824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6227.898269] Lustre: lustre-OST0001: new disk, initializing [ 6227.901194] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6227.905913] Lustre: 196473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6227.968399] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6233.383679] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6235.212445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6235.219309] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6235.268287] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6242.963637] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6245.859873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6245.863360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6245.883545] Lustre: Skipped 2 previous similar messages [ 6245.894078] Lustre: Skipped 1 previous similar message [ 6251.706323] Lustre: server umount lustre-OST0000 complete [ 6253.906435] LustreError: 195451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6257.834567] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6258.033565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6262.194843] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6262.362951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6281.699475] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6281.709442] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6281.711809] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6281.739548] Lustre: Skipped 5 previous similar messages [ 6283.482115] Lustre: server umount lustre-OST0001 complete [ 6289.889703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6289.893328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.900044] Lustre: Skipped 1 previous similar message [ 6294.288239] Lustre: server umount lustre-MDT0000 complete [ 6297.057076] LustreError: 193628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.077945] LustreError: 193628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 6298.002369] LustreError: 196472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446327 with bad export cookie 15892280700255267746 [ 6298.014885] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6298.021491] LustreError: 196472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6298.294613] Lustre: server umount lustre-MDT0001 complete [ 6324.408853] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:39:11 (1788446351) [ 6331.393454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.618030] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6331.746247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6332.034919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6337.250651] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6346.013633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6346.185863] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6346.229927] Lustre: 199914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6350.856376] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6358.419974] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6363.072974] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6369.606423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6369.774889] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6375.524731] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6375.617619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6382.640269] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6390.306060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6390.502223] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6397.380268] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6406.208858] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6411.233882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6411.240297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6411.254369] Lustre: Skipped 1 previous similar message [ 6411.263557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6411.270184] Lustre: Skipped 3 previous similar messages [ 6415.379244] Lustre: server umount lustre-OST0000 complete [ 6416.880401] LustreError: 201762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6432.224679] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6432.227380] LustreError: Skipped 1 previous similar message [ 6437.306302] Lustre: server umount lustre-OST0001 complete [ 6444.000922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.013050] Lustre: Skipped 3 previous similar messages [ 6444.031553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6444.043017] Lustre: Skipped 6 previous similar messages [ 6455.263158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6455.445489] Lustre: server umount lustre-MDT0000 complete [ 6459.086886] LustreError: 199906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446488 with bad export cookie 15892280700255269153 [ 6459.091334] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6459.095698] LustreError: 199906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6459.391774] Lustre: server umount lustre-MDT0001 complete [ 6483.113938] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:41:50 (1788446510) [ 6500.059229] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 6510.571838] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 6544.573135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6544.916813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6544.949379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6545.035427] Lustre: lustre-MDT0000: new disk, initializing [ 6545.171753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.178146] Lustre: Skipped 3 previous similar messages [ 6545.198601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6549.510938] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6561.136631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6561.326885] Lustre: 208011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6561.376269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6561.379939] Lustre: Skipped 1 previous similar message [ 6561.421039] Lustre: lustre-MDT0001: new disk, initializing [ 6561.564678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6561.587527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6567.153869] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6576.384970] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6582.111671] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6589.017610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6589.215982] Lustre: lustre-OST0000: new disk, initializing [ 6589.219276] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6589.224127] Lustre: 209979:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6595.695473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6595.720966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6595.812403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6595.932833] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6605.051481] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6610.106336] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6610.340881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6614.955178] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6615.182233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6621.675352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.689444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6621.702632] Lustre: Skipped 2 previous similar messages [ 6621.703085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6621.707179] Lustre: Skipped 4 previous similar messages [ 6621.709986] LustreError: Skipped 1 previous similar message [ 6623.812890] Lustre: server umount lustre-OST0000 complete [ 6633.309232] Lustre: server umount lustre-MDT0000 complete [ 6637.025241] LustreError: 208023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6637.041054] LustreError: 208023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 6637.779733] LustreError: 208003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446666 with bad export cookie 15892280700255270714 [ 6637.781646] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6637.792441] LustreError: 208003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6638.278663] Lustre: server umount lustre-MDT0001 complete [ 6648.291354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.973150] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6663.096349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6669.369762] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6677.169160] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6684.067784] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6690.685093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6690.957742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6690.961776] Lustre: Skipped 4 previous similar messages [ 6697.037525] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6704.258718] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6708.192149] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6708.428606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6712.970410] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6713.165724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6718.949574] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6718.971487] LustreError: Skipped 1 previous similar message [ 6729.183647] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6729.323405] Lustre: server umount lustre-OST0000 complete [ 6735.436648] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6739.777866] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6768.794129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6774.479878] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6781.719180] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6785.448746] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6785.463937] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6785.467975] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6787.647447] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 6795.751652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6795.760680] LustreError: Skipped 1 previous similar message [ 6795.761651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6795.763686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6795.771104] Lustre: Skipped 5 previous similar messages [ 6795.781232] Lustre: Skipped 10 previous similar messages [ 6798.100573] Lustre: server umount lustre-OST0000 complete [ 6808.719639] Lustre: server umount lustre-MDT0000 complete [ 6812.436348] LustreError: 212934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446841 with bad export cookie 15892280700255271596 [ 6812.439617] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6812.450851] LustreError: 212934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6812.741410] Lustre: server umount lustre-MDT0001 complete [ 6821.757438] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:47:29 (1788446849) [ 6828.031571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.488611] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6840.567104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.182489] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6852.723536] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6857.451604] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6864.191705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6872.050840] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6880.476506] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6884.938540] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6885.086864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6889.142289] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6889.342110] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6895.367281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6895.567916] Lustre: lustre-OST0001: new disk, initializing [ 6895.571868] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6895.580151] Lustre: 221538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6901.964761] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6903.448261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6903.460753] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6903.586720] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6911.717940] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6916.321485] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6916.559647] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6920.764735] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6920.934463] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6924.256399] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6928.804920] Lustre: server umount lustre-OST0001 complete [ 6929.384977] LustreError: 219991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.415609] LustreError: 219991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6935.101596] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6935.237395] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6939.437761] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6939.683734] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6968.241819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6968.382361] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6968.385162] Lustre: Skipped 5 previous similar messages [ 6973.422840] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6973.434152] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6973.434576] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6973.453362] Lustre: Skipped 2 previous similar messages [ 6973.670463] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 6979.917303] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6981.329607] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6989.260173] Lustre: server umount lustre-OST0001 complete [ 7011.383996] Lustre: server umount lustre-OST0000 complete [ 7029.216096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7029.472659] Lustre: server umount lustre-MDT0000 complete [ 7032.908545] LustreError: 224683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447062 with bad export cookie 15892280700255272562 [ 7032.913840] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7032.917718] LustreError: 224683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7039.539235] Lustre: server umount lustre-MDT0001 complete [ 7064.741963] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:51:31 (1788447091) [ 7072.197461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.761165] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7072.844052] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7079.033565] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7086.365899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7086.495983] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7086.540837] Lustre: 226785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7091.839082] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7100.078417] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7105.689705] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7113.250234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7113.485301] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7121.141871] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7130.385482] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7141.562788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7141.677570] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7149.430354] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7156.548241] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7158.096156] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7162.430256] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7162.614211] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7166.510376] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7166.699529] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7171.762844] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97ba186bb800.ost_server_uuid 50 [ 7173.117875] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff97ba186bb800.ost_server_uuid in FULL state after 0 sec [ 7174.092141] Lustre: Permanently deactivating lustre-OST0001 [ 7174.113450] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7176.841471] Lustre: Permanently reactivating lustre-OST0001 [ 7185.379276] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7185.381865] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.391016] LustreError: Skipped 7 previous similar messages [ 7185.400098] Lustre: Skipped 10 previous similar messages [ 7185.406381] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7185.409240] Lustre: Skipped 18 previous similar messages [ 7186.618676] Lustre: server umount lustre-OST0001 complete [ 7196.935553] Lustre: server umount lustre-OST0000 complete [ 7207.691329] Lustre: server umount lustre-MDT0000 complete [ 7210.738321] LustreError: 226777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447239 with bad export cookie 15892280700255273850 [ 7210.751162] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7210.982682] Lustre: server umount lustre-MDT0001 complete [ 7237.711152] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:54:24 (1788447264) [ 7246.662071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.954741] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7246.981750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7246.989284] Lustre: Skipped 5 previous similar messages [ 7251.754826] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7260.925525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.001709] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7261.038926] Lustre: 233475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7265.982181] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7273.826988] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7279.102609] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7284.990924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7285.130361] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7285.139226] Lustre: Permanently deactivating lustre-OST0000 [ 7285.145828] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7290.718424] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7294.501159] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7298.240592] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7305.436972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7305.581587] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7311.941072] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7314.576613] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7319.472447] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7321.230618] Lustre: Permanently reactivating lustre-OST0000 [ 7321.237196] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7321.256653] Lustre: Skipped 2 previous similar messages [ 7322.808827] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7322.819508] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7322.845760] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7322.925094] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7322.976679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7339.172431] Lustre: server umount lustre-OST0001 complete [ 7348.637233] Lustre: server umount lustre-OST0000 complete [ 7358.873184] Lustre: server umount lustre-MDT0000 complete [ 7362.633528] LustreError: 233468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447391 with bad export cookie 15892280700255275502 [ 7362.635251] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7362.641032] LustreError: 233468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7362.873638] Lustre: server umount lustre-MDT0001 complete [ 7374.481715] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 7376.805144] Key type lgssc unregistered [ 7377.079357] LNet: 238359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7377.094849] LNetError: 238359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7377.118825] LNet: Removed LNI 192.168.201.158@tcp [ 7377.904348] Key type .llcrypt unregistered [ 7377.906395] Key type ._llcrypt unregistered [ 7400.498188] Key type ._llcrypt registered [ 7400.500584] Key type .llcrypt registered [ 7400.613578] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7401.510553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7401.742074] alg: No test for adler32 (adler32-zlib) [ 7402.742454] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7402.993463] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 7404.655251] Key type lgssc registered [ 7405.846201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7415.285748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7415.315095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7416.797612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7421.333375] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7426.024158] LustreError: 239825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7426.055903] LustreError: 239825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7430.470824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7430.559768] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7430.604154] Lustre: 239814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7430.615386] Lustre: Permanently deactivating lustre-MDT0001 [ 7430.631893] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7431.035354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7431.660579] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7431.685908] LustreError: 239857:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7431.726143] LustreError: 239857:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7431.755976] LustreError: 239857:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7436.450470] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7445.113022] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7452.132172] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7459.615876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7459.922412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.974409] LustreError: 241669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7460.984807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7466.177528] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7466.473829] LustreError: 241669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.590503] LustreError: 241669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7473.536255] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7475.690493] LustreError: 241754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7475.701727] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7475.705177] LustreError: 241754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7479.451479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.695903] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7484.963803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7484.986932] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7485.792528] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7494.463737] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7498.892611] Lustre: Permanently reactivating lustre-MDT0001 [ 7498.922588] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7498.929181] Lustre: Skipped 2 previous similar messages [ 7500.755675] Lustre: lustre-MDT0001: Client 9cc3b027-f124-4722-bf54-aef49ef7cc95 (at 192.168.201.58@tcp) reconnecting [ 7508.354528] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7509.573827] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7514.956417] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7515.085747] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7519.211376] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7519.417756] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7521.319822] Lustre: Permanently deactivating lustre-MDT0001 [ 7521.335247] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7521.340276] Lustre: Skipped 2 previous similar messages [ 7527.027416] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7538.660708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7538.671879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7538.712983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7542.259259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7542.264575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.298829] Lustre: Skipped 1 previous similar message [ 7543.922239] Lustre: server umount lustre-MDT0000 complete [ 7547.361375] LustreError: 239826:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.399505] LustreError: 239826:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7548.167047] LustreError: 239806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447577 with bad export cookie 7307746967755763071 [ 7548.183112] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7548.183469] LustreError: 239806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7548.643805] Lustre: server umount lustre-MDT0001 complete [ 7554.999409] Lustre: server umount lustre-OST0000 complete [ 7568.905168] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:59:55 (1788447595) [ 7579.822107] Lustre: server umount lustre-OST0001 complete [ 7587.033890] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 7597.810109] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7634.810197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7634.997730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7635.004973] Lustre: Skipped 2 previous similar messages [ 7635.024182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7635.086691] Lustre: lustre-MDT0000: new disk, initializing [ 7635.174463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7635.187909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7639.836835] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7649.855222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7649.962376] Lustre: 248117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7649.989271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7649.992748] Lustre: Skipped 1 previous similar message [ 7650.057826] Lustre: lustre-MDT0001: new disk, initializing [ 7650.155242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7650.190041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7650.205141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7654.488561] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7663.662750] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7668.830651] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7674.718338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.914144] Lustre: lustre-OST0000: new disk, initializing [ 7674.916692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7674.921413] Lustre: 250087:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7674.990562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7680.766673] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7682.624995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7682.640142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7682.739653] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7689.667544] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7691.698221] LustreError: 249749:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7700.688464] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7700.852719] Lustre: lustre-OST0001: new disk, initializing [ 7700.856534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7700.861824] Lustre: 251200:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7701.711142] LustreError: 249749:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7701.726845] LustreError: 248122:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7708.556303] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7709.809789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7709.823451] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7709.895073] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7711.746881] LustreError: 248122:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7717.034868] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7718.548933] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7720.428420] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.432369] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7720.443244] Lustre: Skipped 1 previous similar message [ 7720.445801] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7720.453643] LustreError: Skipped 1 previous similar message [ 7720.459065] Lustre: Skipped 3 previous similar messages [ 7725.545089] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7725.561558] Lustre: Skipped 3 previous similar messages [ 7726.414841] Lustre: server umount lustre-OST0001 complete [ 7727.439786] LustreError: 250067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7744.485701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7744.489971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7744.491122] LustreError: Skipped 1 previous similar message [ 7744.492226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7744.530264] Lustre: Skipped 2 previous similar messages [ 7748.838916] Lustre: server umount lustre-OST0000 complete [ 7752.672682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7752.676566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7752.694856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7752.697060] Lustre: Skipped 3 previous similar messages [ 7758.787726] Lustre: server umount lustre-MDT0000 complete [ 7761.380766] LustreError: 248127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.413488] LustreError: 248127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7762.258158] LustreError: 250084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447791 with bad export cookie 7307746967755765885 [ 7762.267167] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.276232] LustreError: 250084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7768.747459] Lustre: server umount lustre-MDT0001 complete [ 7780.559445] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 7783.561767] Key type lgssc unregistered [ 7783.818862] LNet: 253236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7783.827548] LNetError: 253236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7783.859248] LNet: Removed LNI 192.168.201.158@tcp [ 7784.796240] Key type .llcrypt unregistered [ 7784.799369] Key type ._llcrypt unregistered [ 7826.990457] Key type ._llcrypt registered [ 7826.992940] Key type .llcrypt registered [ 7827.084163] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 7828.238107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7828.298265] alg: No test for adler32 (adler32-zlib) [ 7829.371593] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7829.598875] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 7831.240769] Key type lgssc registered [ 7832.413515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7839.667555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7839.680885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7840.954138] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7840.974405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7841.222706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7845.532961] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7854.368204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7854.451882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7854.479864] Lustre: 255446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7854.707111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7858.630247] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7866.148442] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7872.146134] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7879.988879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7880.172753] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7880.376574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7885.882617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7886.076904] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7893.268625] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7952.361069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7952.364900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7952.388136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7955.290371] Lustre: server umount lustre-OST0000 complete [ 7958.977784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7971.236686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7971.663894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.646597] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7972.679283] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7972.682216] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7972.709798] Lustre: Skipped 1 previous similar message [ 7979.888234] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 7986.789469] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8029.159418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8029.170546] LustreError: Skipped 1 previous similar message [ 8029.175298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.197419] Lustre: Skipped 1 previous similar message [ 8029.205266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8029.211636] Lustre: Skipped 2 previous similar messages [ 8032.742703] Lustre: server umount lustre-OST0000 complete [ 8038.880747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8038.886183] LustreError: Skipped 1 previous similar message [ 8038.891568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8038.911018] Lustre: Skipped 1 previous similar message [ 8038.915072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8043.169178] Lustre: server umount lustre-MDT0000 complete [ 8047.494036] LustreError: 255436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448076 with bad export cookie 3633738774953783950 [ 8047.501023] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8047.512891] LustreError: 255436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8047.960433] Lustre: server umount lustre-MDT0001 complete [ 8060.915609] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 8064.401782] Key type lgssc unregistered [ 8064.734737] LNet: 260870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8064.751851] LNetError: 260870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8064.775563] LNet: Removed LNI 192.168.201.158@tcp [ 8065.663281] Key type .llcrypt unregistered [ 8065.665587] Key type ._llcrypt unregistered [ 8090.213137] Key type ._llcrypt registered [ 8090.214860] Key type .llcrypt registered [ 8090.292561] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8091.319149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8091.401421] alg: No test for adler32 (adler32-zlib) [ 8092.429211] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8092.622936] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 8094.263302] Key type lgssc registered [ 8095.206433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8100.964089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8100.974596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8102.342094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8106.401880] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8112.614291] LustreError: 262290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8112.629232] LustreError: 262290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8114.378581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8114.747227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8118.632571] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8126.047360] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8132.796826] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8141.026686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8141.274513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8146.388825] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8147.441146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8153.664220] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8180.198880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8180.199123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8180.201261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8180.214457] Lustre: Skipped 1 previous similar message [ 8185.317728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8185.333028] Lustre: Skipped 2 previous similar messages [ 8185.739056] Lustre: server umount lustre-OST0000 complete [ 8190.434498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.440385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8190.450973] Lustre: Skipped 1 previous similar message [ 8190.468728] Lustre: Skipped 1 previous similar message [ 8195.554124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8195.558189] Lustre: Skipped 1 previous similar message [ 8200.672812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8200.684239] Lustre: Skipped 1 previous similar message [ 8204.255251] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8204.367204] Lustre: server umount lustre-MDT0000 complete [ 8205.792222] LustreError: 262290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8205.808720] LustreError: 262290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8207.569806] LustreError: 262270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448236 with bad export cookie 15980984005578892370 [ 8207.571055] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8207.577902] LustreError: 262270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8207.918906] Lustre: server umount lustre-MDT0001 complete [ 8219.583407] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 8222.097764] Key type lgssc unregistered [ 8222.396536] LNet: 266603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8222.402541] LNetError: 266603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8222.430379] LNet: Removed LNI 192.168.201.158@tcp [ 8223.188854] Key type .llcrypt unregistered [ 8223.191574] Key type ._llcrypt unregistered [ 8243.410527] Key type ._llcrypt registered [ 8243.413416] Key type .llcrypt registered [ 8243.599622] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8244.896569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8244.945604] alg: No test for adler32 (adler32-zlib) [ 8246.034458] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8246.401035] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 8248.103286] Key type lgssc registered [ 8249.320072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8256.648248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8256.665836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8258.506276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8263.733859] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8268.772664] LustreError: 267825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8268.796411] LustreError: 267825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8272.418795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.763375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8277.119817] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8284.563651] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8291.116646] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8297.438895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8297.773564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8303.512272] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8303.916913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8311.394117] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8327.652177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8327.659273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8327.691797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8332.114824] Lustre: server umount lustre-OST0000 complete [ 8337.888747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8337.897639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8337.912890] Lustre: Skipped 2 previous similar messages [ 8337.928974] Lustre: Skipped 2 previous similar messages [ 8342.591855] Lustre: server umount lustre-MDT0000 complete [ 8343.008395] LustreError: 268552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8343.038597] LustreError: 268552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8347.109194] LustreError: 267824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8347.141851] LustreError: 267824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8347.959354] LustreError: 267806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448377 with bad export cookie 13554347343332295391 [ 8347.973239] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8348.456051] Lustre: server umount lustre-MDT0001 complete [ 8362.730276] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 8365.534540] Key type lgssc unregistered [ 8365.763840] LNet: 271506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8365.769476] LNetError: 271506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8365.782256] LNet: Removed LNI 192.168.201.158@tcp [ 8366.503225] Key type .llcrypt unregistered [ 8366.505193] Key type ._llcrypt unregistered [ 8394.602517] Key type ._llcrypt registered [ 8394.608406] Key type .llcrypt registered [ 8394.698655] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8395.738931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8395.880453] alg: No test for adler32 (adler32-zlib) [ 8397.069926] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8397.409927] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 8399.111203] Key type lgssc registered [ 8400.350749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8409.335800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8409.349832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8410.717470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8415.404383] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8420.832652] LustreError: 272927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.846484] LustreError: 272927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8423.777671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8424.173315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8428.554729] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8436.408292] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8442.126294] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8447.944465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8448.220879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8454.559404] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8455.432730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8462.109345] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8485.345680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8485.352984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8485.364363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8486.370157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8486.377087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8490.664746] Lustre: server umount lustre-OST0000 complete [ 8496.096766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8496.102139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.112738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8496.115623] Lustre: Skipped 1 previous similar message [ 8500.497625] Lustre: server umount lustre-MDT0000 complete [ 8501.728875] LustreError: 272924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8501.757273] LustreError: 272924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8503.802832] LustreError: 272908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448532 with bad export cookie 10215296736385463067 [ 8503.812192] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8503.818738] LustreError: 272908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8504.233711] Lustre: server umount lustre-MDT0001 complete [ 8515.127784] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 8517.532273] Key type lgssc unregistered [ 8517.786676] LNet: 277237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8517.792498] LNetError: 277237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8517.806557] LNet: Removed LNI 192.168.201.158@tcp [ 8518.413226] Key type .llcrypt unregistered [ 8518.414496] Key type ._llcrypt unregistered [ 8535.170500] Key type ._llcrypt registered [ 8535.173141] Key type .llcrypt registered [ 8535.273668] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8536.506657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8536.531755] alg: No test for adler32 (adler32-zlib) [ 8537.630387] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8537.943782] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 8539.671181] Key type lgssc registered [ 8540.751064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8548.093439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8548.117340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8549.636745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8553.808533] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8559.075339] LustreError: 278475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8559.099568] LustreError: 278475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8562.015738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8562.270160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8566.305702] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8573.150649] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8577.888968] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8584.864328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8585.185722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8590.903683] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8594.428423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8598.190906] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8609.765215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8609.769915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8609.770992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8609.786787] Lustre: Skipped 1 previous similar message [ 8609.810284] Lustre: Skipped 1 previous similar message [ 8611.271861] Lustre: server umount lustre-OST0000 complete [ 8618.465385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8618.472129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8618.505338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8620.003571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8620.004080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8620.025836] Lustre: Skipped 1 previous similar message [ 8621.416179] Lustre: server umount lustre-MDT0000 complete [ 8623.008446] LustreError: 278458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8623.025743] LustreError: 278458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8625.612673] LustreError: 278440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448654 with bad export cookie 2918867428583138855 [ 8625.622626] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8625.627478] LustreError: 278440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8625.867407] Lustre: server umount lustre-MDT0001 complete [ 8638.284760] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 8641.541533] Key type lgssc unregistered [ 8641.851892] LNet: 282176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8641.856949] LNetError: 282176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8642.928879] LNet: Removed LNI 192.168.201.158@tcp [ 8644.150372] Key type .llcrypt unregistered [ 8644.151655] Key type ._llcrypt unregistered [ 8656.306905] Key type ._llcrypt registered [ 8656.309118] Key type .llcrypt registered [ 8656.441501] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8672.465348] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 8686.643566] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8687.537737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8687.706493] alg: No test for adler32 (adler32-zlib) [ 8688.735944] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8688.918608] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 8690.655314] Key type lgssc registered [ 8691.744450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8725.362567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8725.377125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8726.645573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8726.674166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8726.796338] Lustre: lustre-MDT0000: new disk, initializing [ 8726.899757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8726.919730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8731.545305] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8741.115760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8741.200479] Lustre: 285728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8741.239856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8741.243712] Lustre: Skipped 1 previous similar message [ 8741.330668] Lustre: lustre-MDT0001: new disk, initializing [ 8741.445321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8741.537199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8741.550395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8745.939968] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8755.105991] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8761.105491] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8767.821955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8768.097499] Lustre: lustre-OST0000: new disk, initializing [ 8768.101147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8768.107953] Lustre: 287697:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8768.206171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.697272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8773.703321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8773.816354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8774.881199] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8784.567499] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8789.639621] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8789.906965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8795.681330] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8795.897180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8799.720216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8799.734795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8799.741644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8799.773145] Lustre: Skipped 1 previous similar message [ 8804.006957] Lustre: server umount lustre-OST0000 complete [ 8807.916596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8807.926088] LustreError: Skipped 1 previous similar message [ 8807.929762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8807.937974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8807.946522] Lustre: Skipped 1 previous similar message [ 8809.952419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.953781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8809.981032] Lustre: Skipped 1 previous similar message [ 8815.075734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8815.080761] Lustre: Skipped 1 previous similar message [ 8819.183021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8819.192653] Lustre: Skipped 1 previous similar message [ 8822.239177] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8822.396897] Lustre: server umount lustre-MDT0000 complete [ 8826.421293] LustreError: 285720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448855 with bad export cookie 1830141524503651337 [ 8826.426648] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8826.431860] LustreError: 285720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8826.793092] Lustre: server umount lustre-MDT0001 complete [ 8836.416376] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:21:03 (1788448863) [ 8842.502633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8842.948190] LustreError: 290109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8842.968854] LustreError: 290109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8843.026249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8847.161080] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8848.360405] LustreError: 290110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8848.390711] LustreError: 290110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8853.475051] LustreError: 290109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8855.386772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8855.786219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8860.954737] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8869.199777] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8876.202871] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8882.731070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8883.141336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8889.492862] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 8899.634320] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8910.819443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8910.826584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8910.827224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8910.857661] Lustre: Skipped 1 previous similar message [ 8916.488198] Lustre: server umount lustre-OST0000 complete [ 8921.057638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.065775] Lustre: Skipped 1 previous similar message [ 8931.300636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8931.313221] Lustre: Skipped 8 previous similar messages [ 8934.367137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8934.593120] Lustre: server umount lustre-MDT0000 complete [ 8936.426584] LustreError: 290109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.449034] LustreError: 290109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8938.206872] LustreError: 290090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448967 with bad export cookie 1830141524503652219 [ 8938.210568] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8938.215159] LustreError: 290090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8938.586590] Lustre: server umount lustre-MDT0001 complete [ 8950.188769] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 8952.385862] Key type lgssc unregistered [ 8952.653696] LNet: 293650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8952.658444] LNetError: 293650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8952.670914] LNet: Removed LNI 192.168.201.158@tcp [ 8953.503742] Key type .llcrypt unregistered [ 8953.511053] Key type ._llcrypt unregistered [ 8979.250422] Key type ._llcrypt registered [ 8979.254891] Key type .llcrypt registered [ 8979.359431] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 8997.398897] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 8998.716780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8998.808463] alg: No test for adler32 (adler32-zlib) [ 9000.397927] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9000.611636] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 9002.247267] Key type lgssc registered [ 9003.229566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9048.869601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9048.942190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.194923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9050.251694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9050.368116] Lustre: lustre-MDT0000: new disk, initializing [ 9050.441394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9050.458259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9054.758842] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9066.645485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.879899] Lustre: 297483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9067.059187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9067.064547] Lustre: Skipped 1 previous similar message [ 9067.186790] Lustre: lustre-MDT0001: new disk, initializing [ 9067.267904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9067.298986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9067.315584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9072.697529] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9084.676464] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9090.605197] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9097.068378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9097.354335] Lustre: lustre-OST2710: new disk, initializing [ 9097.359591] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9097.365852] Lustre: 299454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9097.469398] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9102.958818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9102.972934] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9103.081855] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9103.275787] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9111.791618] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9118.593146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9118.786921] Lustre: lustre-OST03e8: new disk, initializing [ 9118.791709] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9118.797790] Lustre: 300519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9118.883207] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9125.685426] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9127.565096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9127.574591] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9127.684044] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9135.002365] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9143.617680] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9143.903559] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9147.819760] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9148.021882] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9151.518932] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9151.743980] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9155.836341] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9155.994588] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9162.615897] Lustre: server umount lustre-MDT0000 complete [ 9163.744742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9163.760559] Lustre: Skipped 1 previous similar message [ 9165.639943] LustreError: 297477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449194 with bad export cookie 3745664604572336095 [ 9165.647225] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9165.655459] LustreError: 297477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9165.913856] Lustre: server umount lustre-MDT0001 complete [ 9180.415664] Lustre: server umount lustre-OST2710 complete [ 9194.731328] Lustre: server umount lustre-OST03e8 complete [ 9215.651046] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [ 9227.483808] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9265.790301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.967815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9265.985153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9266.047028] Lustre: lustre-MDT0000: new disk, initializing [ 9266.124343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9266.191564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9270.429742] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9280.483422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9280.611879] Lustre: 305772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9280.654109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9280.656936] Lustre: Skipped 1 previous similar message [ 9280.708395] Lustre: lustre-MDT0001: new disk, initializing [ 9280.803246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9280.829168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9280.843302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9284.677628] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9293.160417] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9298.782140] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9305.774141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9305.982468] Lustre: lustre-OST0000: new disk, initializing [ 9305.985244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9305.990256] Lustre: 307741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9306.089382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9312.372184] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9313.391878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9313.412657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9313.575691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9323.267912] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9329.271469] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9329.543854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9334.643236] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9334.862798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9339.360690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9339.364282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.373502] Lustre: Skipped 2 previous similar messages [ 9339.377421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9342.808407] Lustre: server umount lustre-OST0000 complete [ 9347.554305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9347.559504] LustreError: Skipped 1 previous similar message [ 9347.561862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9347.571292] Lustre: Skipped 1 previous similar message [ 9347.576451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9347.578674] Lustre: Skipped 1 previous similar message [ 9349.606478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9349.625140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9352.575487] Lustre: server umount lustre-MDT0000 complete [ 9354.720723] LustreError: 305781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9354.738150] LustreError: 305781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9355.931443] LustreError: 305766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449385 with bad export cookie 3745664604572337782 [ 9355.934723] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9355.944364] LustreError: 305766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9356.264850] Lustre: server umount lustre-MDT0001 complete [ 9366.038986] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:29:53 (1788449393) [ 9367.893405] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9370.110556] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:29:57 (1788449397) [ 9374.299532] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9402.637769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.005289] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9403.046223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9403.203974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9408.421339] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9416.956429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.099983] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9417.145481] Lustre: 311598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9421.494946] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9441.712239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9441.881615] Lustre: Denying initial registration attempt from nid 192.168.201.158@tcp, specified as failover [ 9441.887903] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9441.893830] LustreError: 313050:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9441.898680] LustreError: 313050:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9441.903239] LustreError: 313050:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9441.951947] Lustre: server umount lustre-OST0000 complete [ 9441.955937] LustreError: 313050:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9448.419337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9448.441642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9448.449524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9448.453360] Lustre: Skipped 1 previous similar message [ 9453.687755] Lustre: server umount lustre-MDT0000 complete [ 9455.073755] LustreError: 311605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9455.096541] LustreError: 311605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9457.062952] LustreError: 311591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449486 with bad export cookie 3745664604572338664 [ 9457.071329] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9457.305840] Lustre: server umount lustre-MDT0001 complete [ 9467.691442] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 9470.478432] Key type lgssc unregistered [ 9470.802127] LNet: 314173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9470.807847] LNetError: 314173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9470.822108] LNet: Removed LNI 192.168.201.158@tcp [ 9471.630210] Key type .llcrypt unregistered [ 9471.632642] Key type ._llcrypt unregistered [ 9482.330145] Key type ._llcrypt registered [ 9482.331367] Key type .llcrypt registered [ 9482.404506] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9483.091582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9483.212469] alg: No test for adler32 (adler32-zlib) [ 9484.300200] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9484.596086] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 9486.351214] Key type lgssc registered [ 9487.523113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9538.620886] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9548.555954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9548.570406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9550.039596] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9550.052890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9550.189532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9555.722669] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9566.154677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9566.281848] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9566.372891] Lustre: 316940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9566.711117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9571.602386] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9579.205923] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9584.925514] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9603.820086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9603.976993] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9604.113038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9609.125708] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9616.371716] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9619.941121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9619.943253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9619.946634] LustreError: Skipped 1 previous similar message [ 9619.947757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9619.959939] Lustre: Skipped 1 previous similar message [ 9624.500071] Lustre: server umount lustre-OST0000 complete [ 9628.134904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9628.143913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9628.151084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9628.153781] Lustre: Skipped 1 previous similar message [ 9630.183117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9630.191860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9634.147609] Lustre: server umount lustre-MDT0000 complete [ 9637.634895] LustreError: 316932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449666 with bad export cookie 4842379413951350030 [ 9637.640660] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9637.644878] LustreError: 316932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9637.948561] Lustre: server umount lustre-MDT0001 complete [ 9649.076328] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 9651.928369] Key type lgssc unregistered [ 9652.206751] LNet: 320516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9652.216386] LNetError: 320516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9652.234869] LNet: Removed LNI 192.168.201.158@tcp [ 9653.047649] Key type .llcrypt unregistered [ 9653.052336] Key type ._llcrypt unregistered [ 9679.727551] Key type ._llcrypt registered [ 9679.735917] Key type .llcrypt registered [ 9679.922119] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9682.115716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9682.145086] alg: No test for adler32 (adler32-zlib) [ 9684.172833] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9684.726679] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 9686.575418] Key type lgssc registered [ 9688.780533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9700.194752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9700.234210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9701.821600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9706.923378] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9711.073948] LustreError: 321950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9711.105311] LustreError: 321950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9716.192871] LustreError: 321951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9717.375205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.001879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9723.283822] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9731.801825] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9738.904666] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9748.291654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9748.722187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9755.958287] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9765.081810] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9786.210086] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.58@tcp (stopping) [ 9786.870498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.880404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9786.898815] Lustre: Skipped 1 previous similar message [ 9786.930570] Lustre: Skipped 2 previous similar messages [ 9787.274450] Lustre: server umount lustre-MDT0000 complete [ 9791.304093] LustreError: 321945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9792.241872] LustreError: 321931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449821 with bad export cookie 4764068536726970862 [ 9792.268896] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9792.281752] LustreError: 321931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9796.422429] LustreError: 323242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9796.463352] LustreError: 323242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9796.479424] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (stopping) [ 9797.090959] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9797.110712] Lustre: Skipped 1 previous similar message [ 9799.076205] Lustre: server umount lustre-MDT0001 complete [ 9807.155971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9816.974440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9818.207275] LustreError: 325553:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9818.219621] LustreError: 325553:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000ffa177af x1875325446299520/t0(0) o250->MGC192.168.201.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788449847 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9818.238597] LustreError: 325553:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9818.547689] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x421d5f17d139fef1 [ 9818.562793] Lustre: MGC192.168.201.158@tcp: Connection restored to 0@lo (at 0@lo) [ 9818.979208] LustreError: 323785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9819.007986] LustreError: 323785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9819.066722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9819.127326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:257) [ 9823.696890] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9827.240888] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9827.247995] LustreError: 323785:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9827.257918] LustreError: 323785:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9831.379548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.864538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9835.997834] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9836.976215] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9844.070828] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9846.251121] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9852.314850] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9854.224823] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9860.066515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9860.072961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9860.087861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9860.091642] Lustre: Skipped 1 previous similar message [ 9862.632449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9862.654693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9866.008330] Lustre: server umount lustre-OST0000 complete [ 9872.865418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9872.885830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9872.891909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9872.892636] Lustre: Skipped 1 previous similar message [ 9872.914516] Lustre: Skipped 3 previous similar messages [ 9882.083547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9882.086803] Lustre: Skipped 2 previous similar messages [ 9885.151324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9885.365653] Lustre: server umount lustre-MDT0000 complete [ 9887.214802] LustreError: 325573:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9887.235098] LustreError: 325573:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9889.390040] LustreError: 321932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449918 with bad export cookie 4764068536726978289 [ 9889.394327] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9889.407413] LustreError: 321932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9889.890347] Lustre: server umount lustre-MDT0001 complete [ 9901.653587] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [ 9903.893682] Key type lgssc unregistered [ 9904.189651] LNet: 328247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9904.202564] LNetError: 328247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9905.255353] LNet: Removed LNI 192.168.201.158@tcp [ 9905.895675] Key type .llcrypt unregistered [ 9905.898221] Key type ._llcrypt unregistered [ 9931.242358] Key type ._llcrypt registered [ 9931.244846] Key type .llcrypt registered [ 9931.338394] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [ 9932.651546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9932.688298] alg: No test for adler32 (adler32-zlib) [ 9933.850093] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9934.122995] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [ 9935.839177] Key type lgssc registered [ 9936.890664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9944.243208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9944.276479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9945.815993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9950.037907] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9955.304318] LustreError: 329665:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9955.334909] LustreError: 329665:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9959.502792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.881459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9963.933875] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [ 9972.211900] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9979.111951] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9982.675233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9982.681972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9982.685706] Lustre: Skipped 1 previous similar message [ 9984.738601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9984.743602] Lustre: Skipped 2 previous similar messages [ 9989.215213] Lustre: server umount lustre-MDT0000 complete [ 9990.112340] LustreError: 329664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9990.130268] LustreError: 329664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9992.708395] LustreError: 329646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450021 with bad export cookie 7496872459691866318 [ 9992.710619] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9992.714931] LustreError: 329646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9993.041558] Lustre: server umount lustre-MDT0001 complete [10010.844215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10011.081884] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10011.136800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10011.241897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10016.882880] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10024.857406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10024.948863] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10024.987435] Lustre: 332253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10025.191763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10029.268480] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10038.216985] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10045.267832] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10055.020836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10055.150028] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10055.157292] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10055.164639] LustreError: 334137:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10055.170224] LustreError: 334137:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10055.177506] LustreError: 334137:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10055.269712] Lustre: server umount lustre-OST0000 complete [10055.276225] LustreError: 334137:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10065.258819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10065.452757] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10065.589838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10071.246672] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10072.247261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:289) [10079.232304] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10086.154168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10086.257808] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10086.323710] Lustre: lustre-OST0001: new disk, initializing [10086.327283] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10086.332560] Lustre: 335679:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10086.394497] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10091.864535] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10092.667321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10092.694332] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10092.747826] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10100.082349] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10103.268077] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10103.271476] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10103.281713] LustreError: Skipped 1 previous similar message [10103.282354] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10103.328447] Lustre: Skipped 1 previous similar message [10108.415263] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10108.433748] Lustre: Skipped 2 previous similar messages [10113.506190] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10113.515051] Lustre: Skipped 1 previous similar message [10117.087496] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10117.194318] Lustre: server umount lustre-OST0001 complete [10118.637065] LustreError: 335040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10118.658349] LustreError: 335040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10123.234096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10123.241069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10123.256416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10123.761510] LustreError: 334604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10123.779292] LustreError: 334604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10127.224455] Lustre: server umount lustre-OST0000 complete [10132.960381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10132.964912] LustreError: Skipped 1 previous similar message [10132.972391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10132.986948] Lustre: Skipped 1 previous similar message [10137.720380] Lustre: server umount lustre-MDT0000 complete [10139.112300] LustreError: 333072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10139.128942] LustreError: 333072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10141.006379] LustreError: 332246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450170 with bad export cookie 7496872459691866871 [10141.019188] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10141.024642] LustreError: 332246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10141.541595] Lustre: server umount lustre-MDT0001 complete [10151.779347] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [10154.546805] Key type lgssc unregistered [10154.810774] LNet: 337641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10154.815650] LNetError: 337641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10154.824892] LNet: Removed LNI 192.168.201.158@tcp [10155.647237] Key type .llcrypt unregistered [10155.649965] Key type ._llcrypt unregistered [10228.180775] Key type ._llcrypt registered [10228.186247] Key type .llcrypt registered [10228.347864] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [10245.817448] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10247.301297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10247.332831] alg: No test for adler32 (adler32-zlib) [10248.451545] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10248.677613] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [10250.343234] Key type lgssc registered [10251.464837] Lustre: Echo OBD driver; http://www.lustre.org/ [10287.090951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10287.167756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10288.380482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10288.410036] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10288.563683] Lustre: lustre-MDT0000: new disk, initializing [10288.695853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10288.721110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10293.034445] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10303.654241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10303.886405] Lustre: 342574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10304.008685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10304.016437] Lustre: Skipped 1 previous similar message [10304.137842] Lustre: lustre-MDT0001: new disk, initializing [10304.245482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10304.285717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10304.307586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10310.006203] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10320.417598] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10327.476712] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10333.938987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10334.116480] Lustre: lustre-OST0000: new disk, initializing [10334.119907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10334.125853] Lustre: 344543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10334.196847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10339.423061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10339.446808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10339.481486] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10339.599278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10348.269670] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10352.836305] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10353.001435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10356.756632] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10357.017324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10360.300032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10360.300976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10360.332713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10360.342285] Lustre: Skipped 1 previous similar message [10365.304696] Lustre: server umount lustre-OST0000 complete [10370.530390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10370.533299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10370.540177] Lustre: Skipped 2 previous similar messages [10370.545829] Lustre: Skipped 1 previous similar message [10375.651195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10375.659925] Lustre: Skipped 1 previous similar message [10380.770328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10383.327147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10383.587737] Lustre: server umount lustre-MDT0000 complete [10387.200418] LustreError: 342565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450416 with bad export cookie 5112207096547447108 [10387.211096] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10387.214565] LustreError: 342565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10387.681395] Lustre: server umount lustre-MDT0001 complete [10396.696544] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:47:04 (1788450424) [10404.421776] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:47:11 (1788450431) [10412.279157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10412.769038] LustreError: 347299:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10412.793119] LustreError: 347299:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10412.845972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10417.305841] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10418.152741] LustreError: 347300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10422.241519] LustreError: 347300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.538964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10424.773235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10428.493633] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10435.144136] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10440.357797] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10446.974571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10447.357799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10453.395189] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10461.149944] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10469.968925] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10471.797525] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10473.464841] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10475.105310] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10476.848981] Lustre: Failing over lustre-MDT0000 [10477.048251] Lustre: server umount lustre-MDT0000 complete [10478.057864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10478.065783] LustreError: 347300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10478.069392] Lustre: Skipped 2 previous similar messages [10478.088465] LustreError: 347300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10483.169340] LustreError: 347294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10483.210316] LustreError: 347294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10493.409047] LustreError: 347300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10493.444039] LustreError: 347300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10494.438939] Lustre: 340800:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450507/real 1788450507] req@00000000b4729626 x1875326037724672/t0(0) o400->MGC192.168.201.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788450523 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10494.472464] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10494.703354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10504.674897] LustreError: 340797:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ca892f9c x1875326037730816/t0(0) o250->MGC192.168.201.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10505.117456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10505.236934] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10510.156549] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10510.312961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10510.340866] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10510.412081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10510.549511] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10522.097822] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10524.126256] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10527.507152] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10529.472304] Lustre: Failing over lustre-MDT0000 [10529.708616] Lustre: server umount lustre-MDT0000 complete [10530.698933] LustreError: 351132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.717531] LustreError: 351132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [10530.784439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10530.788052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.788847] LustreError: Skipped 1 previous similar message [10530.800864] Lustre: Skipped 2 previous similar messages [10547.227435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.340392] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10547.538227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10547.585233] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10551.697433] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10552.134079] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10552.809838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10552.826583] Lustre: Skipped 2 previous similar messages [10552.858583] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10552.889118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10561.235809] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10562.681588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10573.283885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10573.289665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10573.292921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10573.292929] Lustre: Skipped 1 previous similar message [10573.315191] Lustre: Skipped 2 previous similar messages [10575.344627] Lustre: server umount lustre-MDT0000 complete [10578.402194] LustreError: 348027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10578.418985] LustreError: 348027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [10579.300096] LustreError: 349838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450608 with bad export cookie 5112207096547449824 [10579.307449] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10579.309917] LustreError: 349838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10579.582420] Lustre: server umount lustre-MDT0001 complete [10593.511490] Lustre: server umount lustre-OST0000 complete [10607.163321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10611.714412] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10620.591153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10620.994299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10621.000603] Lustre: Skipped 1 previous similar message [10625.493940] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10632.404248] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10638.124674] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10644.911107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10650.757690] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10652.344654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10658.535488] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10665.970908] Lustre: server umount lustre-OST0000 complete [10672.097921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10672.103414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10672.114512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10672.120721] Lustre: Skipped 2 previous similar messages [10679.325546] Lustre: server umount lustre-MDT0000 complete [10682.916791] LustreError: 353622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450712 with bad export cookie 5112207096547450566 [10682.918049] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10682.926225] LustreError: 353622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10693.389941] Lustre: server umount lustre-MDT0001 complete [10705.564917] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [10708.550216] Key type lgssc unregistered [10708.865578] LNet: 357174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10708.871463] LNetError: 357174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10708.893634] LNet: Removed LNI 192.168.201.158@tcp [10709.847164] Key type .llcrypt unregistered [10709.848538] Key type ._llcrypt unregistered [10735.659875] Key type ._llcrypt registered [10735.662275] Key type .llcrypt registered [10735.764973] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [10737.297512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10737.319095] alg: No test for adler32 (adler32-zlib) [10738.356695] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10738.574650] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [10740.247267] Key type lgssc registered [10741.452423] Lustre: Echo OBD driver; http://www.lustre.org/ [10748.603910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10748.637617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10750.138585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10754.630922] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10760.686924] LustreError: 358693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10760.718178] LustreError: 358693:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10762.108460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10762.622046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10766.924614] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10776.833520] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10783.601229] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10790.864027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10791.139616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10795.248274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10796.965561] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10803.885895] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10819.042244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10819.050401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10819.068599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10820.489109] Lustre: server umount lustre-MDT0000 complete [10823.141692] LustreError: 358693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10823.144575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10823.167578] Lustre: Skipped 1 previous similar message [10823.548536] LustreError: 358674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450852 with bad export cookie 8366326646121989147 [10823.556922] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10823.761508] Lustre: server umount lustre-MDT0001 complete [10836.371483] Lustre: server umount lustre-OST0000 complete [10850.908388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.631195] LustreError: 362426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.670321] LustreError: 362426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10851.807408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10856.807866] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10856.951277] LustreError: 362427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.035111] LustreError: 362426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.424435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.771868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10871.257576] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10879.412310] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10884.874679] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10891.413421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10891.616716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10896.956714] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10897.771954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10904.667167] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10917.857171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10917.877098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10917.892921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10922.303710] Lustre: server umount lustre-MDT0000 complete [10923.490648] LustreError: 362426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10923.520190] LustreError: 362426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10925.926480] LustreError: 362408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450955 with bad export cookie 8366326646121990260 [10925.941083] LustreError: 362408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10925.954770] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10926.291593] Lustre: server umount lustre-MDT0001 complete [10940.703824] Lustre: server umount lustre-OST0000 complete [10955.314435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10955.657812] LustreError: 366333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.734332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10959.407607] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10966.407441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10970.460801] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10976.990805] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10981.909669] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10987.413813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10993.668782] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [10999.924689] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11000.014546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11005.409655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11005.418332] Lustre: Skipped 3 previous similar messages [11005.418657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11005.424714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11005.428297] Lustre: Skipped 3 previous similar messages [11006.386844] Lustre: server umount lustre-OST0000 complete [11010.529486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11010.536780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11010.544522] Lustre: Skipped 1 previous similar message [11010.560051] Lustre: Skipped 1 previous similar message [11015.650911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11015.657425] Lustre: Skipped 1 previous similar message [11024.351182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11024.608570] Lustre: server umount lustre-MDT0000 complete [11025.888319] LustreError: 366334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11025.921300] LustreError: 366334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11028.265589] LustreError: 366313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451057 with bad export cookie 8366326646121991282 [11028.268127] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11028.287512] LustreError: 366313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11028.722180] Lustre: server umount lustre-MDT0001 complete [11038.870631] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [11041.833413] Key type lgssc unregistered [11042.118960] LNet: 369865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11042.135871] LNetError: 369865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11042.162287] LNet: Removed LNI 192.168.201.158@tcp [11042.976953] Key type .llcrypt unregistered [11042.978200] Key type ._llcrypt unregistered [11073.667873] Key type ._llcrypt registered [11073.671585] Key type .llcrypt registered [11073.844453] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11075.415450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11075.476817] alg: No test for adler32 (adler32-zlib) [11076.527704] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11076.745145] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [11078.431177] Key type lgssc registered [11079.244947] Lustre: Echo OBD driver; http://www.lustre.org/ [11086.038030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11086.046281] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11086.052233] LustreError: 371309:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11086.066377] LustreError: 371309:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11086.071456] LustreError: 371309:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11086.076415] LustreError: 371309:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11086.081198] LustreError: 371309:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11094.090336] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11094.101532] LustreError: 371708:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11094.126497] LustreError: 371708:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11094.136833] LustreError: 371708:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11094.143152] LustreError: 371708:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11094.151713] LustreError: 371708:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11114.283561] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [11117.714729] Key type lgssc unregistered [11118.106822] LNet: 372714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11118.111599] LNetError: 372714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11118.122231] LNet: Removed LNI 192.168.201.158@tcp [11118.876877] Key type .llcrypt unregistered [11118.878905] Key type ._llcrypt unregistered [11137.823566] Key type ._llcrypt registered [11137.826384] Key type .llcrypt registered [11137.977675] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [11153.621803] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11154.682875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11154.803030] alg: No test for adler32 (adler32-zlib) [11155.805963] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11155.997874] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [11157.616457] Key type lgssc registered [11158.833030] Lustre: Echo OBD driver; http://www.lustre.org/ [11189.463299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11189.484623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.728717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11190.775374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11190.836233] Lustre: lustre-MDT0000: new disk, initializing [11190.948866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11190.964393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11195.146230] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11203.780677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11203.865476] Lustre: 375871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11203.908831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11203.911926] Lustre: Skipped 1 previous similar message [11203.972328] Lustre: lustre-MDT0001: new disk, initializing [11204.045944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11204.079189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11204.098203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11208.062582] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11216.585232] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11222.271375] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11229.879989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11230.218794] Lustre: lustre-OST0000: new disk, initializing [11230.226899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11230.231791] Lustre: 377838:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11230.301589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11236.424746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11236.430764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11236.554043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11236.833279] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11247.742851] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11252.428615] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11252.738953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11258.670910] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11258.965704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11262.436827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11262.442097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11262.455080] Lustre: Skipped 1 previous similar message [11262.481831] Lustre: Skipped 1 previous similar message [11267.554111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11267.563526] Lustre: Skipped 1 previous similar message [11272.679756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11272.694159] Lustre: Skipped 1 previous similar message [11275.231205] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11275.371683] Lustre: server umount lustre-OST0000 complete [11280.866754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11280.869919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11280.896233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11282.914327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11285.552015] Lustre: server umount lustre-MDT0000 complete [11288.994445] LustreError: 375863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451318 with bad export cookie 2438303918725077419 [11289.007100] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11289.501519] Lustre: server umount lustre-MDT0001 complete [11298.645865] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11300.376807] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:02:07 (1788451327) [11307.385348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11307.823395] LustreError: 380311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11307.839210] LustreError: 380311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11307.891448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11312.731302] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11313.122616] LustreError: 380312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11317.229931] LustreError: 380311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.928728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11322.328541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11327.158693] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11334.768928] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11340.857168] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11347.779395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11347.988653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11353.944427] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11361.007645] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11367.725881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11367.958906] Lustre: lustre-OST0001: new disk, initializing [11367.963864] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11367.981443] Lustre: 383148:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11368.079646] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11374.086779] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11374.735946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11374.751891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11374.840615] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11382.556275] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11389.294331] Lustre: lustre-OST0001: Not available for connect from 192.168.201.58@tcp (stopping) [11389.303980] Lustre: Skipped 2 previous similar messages [11390.439864] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11390.442665] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11390.452474] Lustre: Skipped 1 previous similar message [11390.478942] LustreError: Skipped 1 previous similar message [11392.146207] Lustre: server umount lustre-OST0001 complete [11394.373313] LustreError: 382135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11400.682444] LustreError: 382133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11400.710383] LustreError: 382133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11408.358786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11408.362875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11408.365170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11408.369107] Lustre: Skipped 1 previous similar message [11408.390327] Lustre: Skipped 3 previous similar messages [11410.918046] LustreError: 382134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11410.940896] LustreError: 382134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11414.289717] Lustre: server umount lustre-OST0000 complete [11419.616931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11419.623188] LustreError: Skipped 1 previous similar message [11419.627310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11426.274515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11426.288438] Lustre: Skipped 6 previous similar messages [11431.396208] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11433.623178] Lustre: server umount lustre-MDT0000 complete [11436.514165] LustreError: 381039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.524356] LustreError: 381039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11436.950260] LustreError: 383147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451466 with bad export cookie 2438303918725078301 [11436.958991] LustreError: 383147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11436.963179] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11451.359249] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11451.619735] Lustre: server umount lustre-MDT0001 complete [11462.848572] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [11465.651899] Key type lgssc unregistered [11465.885822] LNet: 385141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11465.891848] LNetError: 385141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11465.911938] LNet: Removed LNI 192.168.201.158@tcp [11466.971449] Key type .llcrypt unregistered [11466.975756] Key type ._llcrypt unregistered [11521.736579] Key type ._llcrypt registered [11521.741702] Key type .llcrypt registered [11521.868418] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11523.443512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11523.481934] alg: No test for adler32 (adler32-zlib) [11524.596761] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11524.832300] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [11526.535276] Key type lgssc registered [11527.772995] Lustre: Echo OBD driver; http://www.lustre.org/ [11536.327176] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11536.338379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11537.526837] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11537.540523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11537.680996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11542.216739] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11552.936557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11553.112654] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11553.217428] Lustre: 387938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11553.580587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11557.862929] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11564.889370] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11570.737698] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11574.241586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11574.262539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11574.277611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11576.800816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11576.802115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11578.726261] Lustre: server umount lustre-MDT0000 complete [11582.442353] LustreError: 387933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451611 with bad export cookie 9850526989120456322 [11582.448816] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11582.451761] LustreError: 387933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11582.637862] Lustre: server umount lustre-MDT0001 complete [11592.689670] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:07:00 (1788451620) [11599.170595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11599.412944] LustreError: 390452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11599.427267] LustreError: 390452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11599.455308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11604.021461] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11604.451841] LustreError: 390453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11609.577383] LustreError: 390452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11612.859053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11613.232925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11618.508572] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11627.731440] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11633.894954] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11640.051194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11640.258589] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11640.382609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11646.858617] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11655.225818] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11662.745153] Lustre: Permanently deactivating lustre-OST0000 [11662.778424] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11672.632508] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11674.779521] LustreError: 393488:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11674.797032] LustreError: 393488:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11674.803154] LustreError: 393488:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11679.002643] LustreError: 393537:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11679.011121] LustreError: 393537:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11687.049115] Lustre: server umount lustre-OST0000 complete [11689.991447] LustreError: 393788:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11689.996309] LustreError: 393788:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11693.026566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11693.040803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11693.041901] Lustre: Skipped 1 previous similar message [11693.053796] Lustre: Skipped 2 previous similar messages [11698.144614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11698.150790] Lustre: Skipped 1 previous similar message [11703.265772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11703.279440] Lustre: Skipped 1 previous similar message [11706.335132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11706.616714] Lustre: server umount lustre-MDT0000 complete [11708.386464] LustreError: 390448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11708.406037] LustreError: 390448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11710.091099] LustreError: 390433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451739 with bad export cookie 9850526989120456896 [11710.092673] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11710.107391] LustreError: 390433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11720.829574] Lustre: server umount lustre-MDT0001 complete [11730.704675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11735.058267] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11737.727889] LustreError: 394947:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11743.434738] Lustre: 395286:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.158@tcp [11745.170703] Lustre: server umount MGS complete [11755.202203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11755.724703] LustreError: 396010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11755.744619] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11755.744627] Lustre: Skipped 1 previous similar message [11755.791849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11760.498422] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11766.251089] LustreError: 396011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.270373] LustreError: 396011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11768.749521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11769.114443] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11773.372337] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11781.225095] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11787.773196] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11789.418589] Lustre: Permanently reactivating lustre-OST0000 [11789.442181] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11789.449592] Lustre: Skipped 2 previous similar messages [11817.440287] LustreError: 397793:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11824.542169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11824.893645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11824.909632] Lustre: Skipped 1 previous similar message [11831.302682] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11838.897738] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11846.621556] Lustre: server umount lustre-OST0000 complete [11851.240098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11851.252246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11851.263906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11860.081377] Lustre: server umount lustre-MDT0000 complete [11863.007203] Lustre: 387264:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451876/real 1788451876] req@00000000b41755aa x1875327375738368/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788451892 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11863.011758] LustreError: 396740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.035508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11863.054428] LustreError: 396740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11863.063740] Lustre: Skipped 1 previous similar message [11863.496372] LustreError: 395991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451892 with bad export cookie 9850526989120457953 [11863.501869] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11863.509835] LustreError: 395991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11864.044539] Lustre: server umount lustre-MDT0001 complete [11878.168319] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [11881.155230] Key type lgssc unregistered [11881.420074] LNet: 400382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11881.429055] LNetError: 400382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11882.473812] LNet: Removed LNI 192.168.201.158@tcp [11883.284495] Key type .llcrypt unregistered [11883.286840] Key type ._llcrypt unregistered [11900.721954] Key type ._llcrypt registered [11900.723510] Key type .llcrypt registered [11900.807386] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [11917.601946] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [11918.992305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11919.065584] alg: No test for adler32 (adler32-zlib) [11920.189422] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11920.457441] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [11922.079191] Key type lgssc registered [11923.149878] Lustre: Echo OBD driver; http://www.lustre.org/ [11958.132466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11958.151940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11959.362835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11959.387689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11959.430156] Lustre: lustre-MDT0000: new disk, initializing [11959.489967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11959.501489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11964.054139] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11975.242883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11975.415941] Lustre: 403536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11975.494260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11975.496837] Lustre: Skipped 1 previous similar message [11975.586653] Lustre: lustre-MDT0001: new disk, initializing [11975.712202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11975.740482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11975.758402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11979.999445] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [11990.583186] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11996.824930] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12004.426796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12004.802510] Lustre: lustre-OST0000: new disk, initializing [12004.808486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12004.822410] Lustre: 405507:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12004.916403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12010.661261] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12011.610202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12011.624195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12011.787408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12020.979318] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12026.162356] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12026.390661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12031.693303] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12031.905476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12037.601210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12037.612344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12037.614726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12037.624909] Lustre: Skipped 1 previous similar message [12037.644658] LustreError: Skipped 1 previous similar message [12039.612412] Lustre: server umount lustre-OST0000 complete [12047.329167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12047.345935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12047.369556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12047.376632] Lustre: Skipped 1 previous similar message [12049.760459] Lustre: server umount lustre-MDT0000 complete [12053.255043] LustreError: 403529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452082 with bad export cookie 5019546220788617657 [12053.258145] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12053.266894] LustreError: 403529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12053.507631] Lustre: server umount lustre-MDT0001 complete [12062.664530] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12064.468437] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:14:51 (1788452091) [12071.738030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12072.180527] LustreError: 407968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.201063] LustreError: 407968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12072.291329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12077.154891] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12077.537130] LustreError: 407969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.657459] LustreError: 407968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12086.296231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12086.781551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12091.323585] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12099.207677] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12104.764625] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12112.564091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12112.792616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12119.866481] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12128.506628] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12130.741523] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12133.617930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12138.466795] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12138.470374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12138.489699] Lustre: Skipped 1 previous similar message [12138.497896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12138.512546] Lustre: Skipped 1 previous similar message [12141.537102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12141.538649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12141.541818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12144.554990] Lustre: server umount lustre-OST0000 complete [12151.778886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12151.793559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12151.803801] Lustre: Skipped 1 previous similar message [12151.813621] Lustre: Skipped 2 previous similar messages [12154.920770] Lustre: server umount lustre-MDT0000 complete [12156.902135] LustreError: 407963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12156.940353] LustreError: 407963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12159.395659] LustreError: 407950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452188 with bad export cookie 5019546220788618539 [12159.401235] LustreError: 407950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12159.401508] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12159.757693] Lustre: server umount lustre-MDT0001 complete [12171.234094] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12174.674886] Key type lgssc unregistered [12175.041514] LNet: 411698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12175.053902] LNetError: 411698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12175.070372] LNet: Removed LNI 192.168.201.158@tcp [12176.081188] Key type .llcrypt unregistered [12176.083676] Key type ._llcrypt unregistered [12205.606122] Key type ._llcrypt registered [12205.610257] Key type .llcrypt registered [12205.738989] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12206.060598] Key type .llcrypt unregistered [12206.062630] Key type ._llcrypt unregistered [12224.264962] Key type ._llcrypt registered [12224.266660] Key type .llcrypt registered [12224.371386] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12225.356561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12225.366747] alg: No test for adler32 (adler32-zlib) [12226.372624] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12226.591194] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12228.287295] Key type lgssc registered [12229.430246] Lustre: Echo OBD driver; http://www.lustre.org/ [12238.678610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12238.694476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12240.177332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12245.922983] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12250.593808] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.620543] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12254.540234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12254.881502] LustreError: 413916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12254.909709] LustreError: 413916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12254.918551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12260.320599] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.343054] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12260.670681] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12265.443500] LustreError: 413916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12265.473658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12265.482489] LustreError: 413916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12269.752283] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12270.572383] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12270.591865] LustreError: 413915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12276.495214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12276.793263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12281.254713] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12291.044349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12291.057899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12291.077927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12291.366292] Lustre: server umount lustre-OST0000 complete [12297.205816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12297.206081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12297.213487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12297.222416] Lustre: Skipped 1 previous similar message [12301.367199] Lustre: server umount lustre-MDT0000 complete [12302.309350] LustreError: 413916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12302.328289] LustreError: 413916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12305.640625] LustreError: 416513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452334 with bad export cookie 802703701101875702 [12305.648049] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12305.657948] LustreError: 416513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12315.327404] Lustre: server umount lustre-MDT0001 complete [12327.889498] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12333.238436] Key type lgssc unregistered [12333.808950] LNet: 417067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12333.847657] LNetError: 417067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12333.880122] LNet: Removed LNI 192.168.201.158@tcp [12335.285073] Key type .llcrypt unregistered [12335.288301] Key type ._llcrypt unregistered [12361.842421] Key type ._llcrypt registered [12361.847508] Key type .llcrypt registered [12361.968974] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12363.587889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12363.632102] alg: No test for adler32 (adler32-zlib) [12364.756876] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12365.030250] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12366.712540] Key type lgssc registered [12367.934469] Lustre: Echo OBD driver; http://www.lustre.org/ [12375.362749] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12375.386375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12390.879774] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12395.999565] LustreError: 418472:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.158@tcp: failed processing log, type 4: rc = -110 [12422.687428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12428.819540] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12438.088402] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12444.814645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12445.581188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12445.640499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12450.777580] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12450.789098] LustreError: 418497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12450.815366] LustreError: 418497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12455.905614] LustreError: 418498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.932494] LustreError: 418497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.960528] LustreError: 418497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12460.320816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12460.741613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12465.921995] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12473.896620] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12481.333295] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12489.830558] Lustre: server umount lustre-OST0000 complete [12494.828676] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12494.843348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12494.860844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12496.864894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12496.865577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12501.985060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12501.994483] Lustre: Skipped 2 previous similar messages [12504.202429] Lustre: server umount lustre-MDT0000 complete [12507.105314] LustreError: 419510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12507.143260] LustreError: 419510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12508.083635] LustreError: 418478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452537 with bad export cookie 17107800998142116514 [12508.089887] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12518.532323] Lustre: server umount lustre-MDT0001 complete [12533.278344] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12536.788971] Key type lgssc unregistered [12537.212360] LNet: 422046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12537.219811] LNetError: 422046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12537.236736] LNet: Removed LNI 192.168.201.158@tcp [12538.155625] Key type .llcrypt unregistered [12538.157709] Key type ._llcrypt unregistered [12564.721014] Key type ._llcrypt registered [12564.722473] Key type .llcrypt registered [12564.824310] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12565.729635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12565.777091] alg: No test for adler32 (adler32-zlib) [12566.843744] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12567.077646] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12568.704299] Key type lgssc registered [12569.775153] Lustre: Echo OBD driver; http://www.lustre.org/ [12578.590390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12578.623033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.985286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12585.067676] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12590.562318] LustreError: 423463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12590.575837] LustreError: 423463:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12593.475083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12593.789828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12598.693448] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12606.557463] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12612.920630] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12620.140242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12620.477642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12625.679327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12628.223319] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12638.434100] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12643.305398] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12643.323192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12643.336973] Lustre: Skipped 2 previous similar messages [12643.353969] Lustre: Skipped 2 previous similar messages [12645.732064] Lustre: server umount lustre-MDT0000 complete [12645.744761] LustreError: 424198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12648.419946] LustreError: 423464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12648.436572] LustreError: 423464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12648.708875] LustreError: 423443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.58@tcp arrived at 1788452677 with bad export cookie 17250981138329188808 [12648.719381] LustreError: 423443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12650.845800] LustreError: 423445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452679 with bad export cookie 17250981138329187905 [12650.884057] LustreError: 423445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12650.893623] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12653.541979] LustreError: 424191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12653.557388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12653.579488] LustreError: 424191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12653.598759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12653.608418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12655.589633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12657.179869] Lustre: server umount lustre-OST0000 complete [12660.707725] LustreError: 424191:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12660.725672] LustreError: 424191:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12665.017709] Lustre: server umount lustre-MDT0001 complete [12679.366791] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12682.358950] Key type lgssc unregistered [12682.617670] LNet: 427151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12682.622422] LNetError: 427151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12682.644393] LNet: Removed LNI 192.168.201.158@tcp [12683.488443] Key type .llcrypt unregistered [12683.497442] Key type ._llcrypt unregistered [12710.010201] Key type ._llcrypt registered [12710.012588] Key type .llcrypt registered [12710.164255] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12711.699485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12711.722099] alg: No test for adler32 (adler32-zlib) [12712.810537] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12713.094968] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12714.783403] Key type lgssc registered [12716.025773] Lustre: Echo OBD driver; http://www.lustre.org/ [12723.618803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12723.639355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12725.049731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12728.779605] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12735.468471] LustreError: 428571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12735.490377] LustreError: 428571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12736.981214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12737.418486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12743.207666] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12751.663295] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12758.709684] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12765.650828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12766.103908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12770.217308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12772.560362] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12781.642264] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12787.688620] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12787.696177] Lustre: Skipped 1 previous similar message [12787.702925] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12789.306658] Lustre: server umount lustre-MDT0001 complete [12790.144313] LustreError: 428567:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12792.801197] LustreError: 428570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12792.813802] LustreError: 428570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12795.872688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12795.879798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12795.921693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12795.937294] Lustre: Skipped 1 previous similar message [12797.920510] LustreError: 428571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.926485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12801.050847] Lustre: server umount lustre-OST0000 complete [12803.041608] LustreError: 429310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12805.806540] Lustre: server umount lustre-MDT0000 complete [12826.167763] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [12831.445431] Key type lgssc unregistered [12832.089450] LNet: 432260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12832.099818] LNetError: 432260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12832.137523] LNet: Removed LNI 192.168.201.158@tcp [12833.864916] Key type .llcrypt unregistered [12833.867983] Key type ._llcrypt unregistered [12861.073554] Key type ._llcrypt registered [12861.075259] Key type .llcrypt registered [12861.261337] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [12883.110356] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12884.179594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12884.274414] alg: No test for adler32 (adler32-zlib) [12885.812917] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12886.464178] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [12888.431543] Key type lgssc registered [12889.879823] Lustre: Echo OBD driver; http://www.lustre.org/ [12934.025457] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [12954.132154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12954.160228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12955.733796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12955.789523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12955.873462] Lustre: lustre-MDT0000: new disk, initializing [12955.975465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12956.000619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12961.118740] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12978.881251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12979.109481] Lustre: lustre-OST0000: new disk, initializing [12979.112782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12979.116286] Lustre: Skipped 1 previous similar message [12979.119929] Lustre: 437295:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12979.186506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12985.293285] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [12988.582109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12988.588254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12988.611084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12996.058747] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13010.861432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13010.951838] Lustre: 436276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13011.010553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13011.065914] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.58@tcp (not set up) [13011.087742] Lustre: lustre-MDT0001: new disk, initializing [13011.183410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13011.212785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13011.219641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13015.211876] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13024.307404] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13024.616661] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13029.741803] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13029.995193] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13036.521354] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13036.531631] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13036.570584] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13036.593119] Lustre: Skipped 1 previous similar message [13041.638162] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13041.649512] Lustre: Skipped 2 previous similar messages [13042.198694] Lustre: server umount lustre-MDT0001 complete [13047.110467] LustreError: 439417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13047.146346] LustreError: 439417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13051.884589] LustreError: 436289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13051.909301] LustreError: 436289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13057.011201] LustreError: 438621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13057.031603] LustreError: 438621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13062.112757] LustreError: 436288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13062.116202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13062.132201] Lustre: Skipped 1 previous similar message [13062.140379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13067.234437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13067.236750] LustreError: 436289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13068.107975] Lustre: server umount lustre-OST0000 complete [13079.424945] Lustre: server umount lustre-MDT0000 complete [13096.004077] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13098.736652] Key type lgssc unregistered [13098.955516] LNet: 441053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13098.963177] LNetError: 441053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13100.007817] LNet: Removed LNI 192.168.201.158@tcp [13100.780346] Key type .llcrypt unregistered [13100.782361] Key type ._llcrypt unregistered [13154.781345] Key type ._llcrypt registered [13154.786952] Key type .llcrypt registered [13154.919176] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [13155.319804] Key type .llcrypt unregistered [13155.329491] Key type ._llcrypt unregistered [13173.075170] Key type ._llcrypt registered [13173.086350] Key type .llcrypt registered [13173.283982] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [13174.502447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13174.719201] alg: No test for adler32 (adler32-zlib) [13175.747226] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13176.029244] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [13177.743192] Key type lgssc registered [13178.814582] Lustre: Echo OBD driver; http://www.lustre.org/ [13208.435761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13208.485711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13209.786298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13209.804704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13209.861766] Lustre: lustre-MDT0000: new disk, initializing [13209.910072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13209.923984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13215.306049] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13226.637696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13226.775214] Lustre: 444435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13226.837642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13226.841726] Lustre: Skipped 1 previous similar message [13226.906792] Lustre: lustre-MDT0001: new disk, initializing [13227.044048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13227.119281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13227.140311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13231.917264] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13240.907668] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13246.634980] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13253.075626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13253.345218] Lustre: lustre-OST0000: new disk, initializing [13253.348787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13253.354120] Lustre: 446403:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13253.451899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13259.401701] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13260.354687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13260.371131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13260.570775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13268.811698] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13276.144325] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13276.151024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13276.159483] Lustre: Skipped 2 previous similar messages [13276.167254] Lustre: Skipped 1 previous similar message [13281.251273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13281.262873] Lustre: Skipped 3 previous similar messages [13281.959780] Lustre: server umount lustre-MDT0000 complete [13285.978648] LustreError: 444428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453315 with bad export cookie 11463092291678136779 [13285.999477] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13286.308977] Lustre: server umount lustre-MDT0001 complete [13300.235825] Lustre: server umount lustre-OST0000 complete [13311.822833] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:35:39 (1788453339) [13318.799965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13319.236808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13323.043297] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13324.263452] LustreError: 448431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13324.293364] LustreError: 448431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13329.397910] LustreError: 448431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13331.695347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13332.293837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13337.866907] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13346.144435] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13352.288446] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13371.759264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13371.996553] Lustre: Found index 0 for lustre-OST0000, updating log [13372.292334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13380.899758] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [13389.352105] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13390.712740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13401.064398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13401.069964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13401.077071] Lustre: Skipped 1 previous similar message [13401.088900] Lustre: Skipped 1 previous similar message [13407.455922] Lustre: server umount lustre-OST0000 complete [13424.609755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13424.630835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13424.658278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13424.674090] Lustre: Skipped 2 previous similar messages [13428.705928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13428.707302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13428.744799] Lustre: Skipped 1 previous similar message [13430.049631] Lustre: server umount lustre-MDT0000 complete [13430.825540] LustreError: 448426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13430.842946] LustreError: 448426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13434.521467] LustreError: 448413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453463 with bad export cookie 11463092291678138172 [13434.530427] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13434.540883] LustreError: 448413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [13434.936861] Lustre: server umount lustre-MDT0001 complete [13439.271717] LNet: 1 local NIs in recovery (showing 1): 192.168.201.158@tcp [13444.070639] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:37:51 (1788453471) [14070.328230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14070.856205] LustreError: 454847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14070.916525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14075.212404] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14076.408523] LustreError: 454848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14080.483751] LustreError: 454847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14084.773340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14085.296985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14091.965256] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14102.077839] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14108.624861] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14129.580369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14129.969033] Lustre: Found index 0 for lustre-OST0000, updating log [14130.325939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14130.540774] LNet: 1 local NIs in recovery (showing 1): 192.168.201.158@tcp [14136.379350] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14144.018871] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14147.170894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14147.205403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14147.215235] Lustre: Skipped 2 previous similar messages [14152.461735] Lustre: server umount lustre-OST0000 complete [14156.770023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14156.790524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14156.826564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14157.860562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14158.884430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14158.894073] Lustre: Skipped 3 previous similar messages [14162.682834] Lustre: server umount lustre-MDT0000 complete [14162.977714] LustreError: 455577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14163.004689] LustreError: 455577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14167.081645] LustreError: 454843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14167.124485] LustreError: 454843:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [14167.180934] LustreError: 454829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454196 with bad export cookie 11463092291678139257 [14167.188973] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14167.216187] LustreError: 454829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14168.303390] Lustre: server umount lustre-MDT0001 complete [14189.919265] LNet: 1 local NIs in recovery (showing 1): 192.168.201.158@tcp [14201.703839] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [14211.751339] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [14246.471295] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:51:13 (1788454273) [14265.656176] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [14268.365202] Key type lgssc unregistered [14268.698959] LNet: 461477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14268.706685] LNetError: 461477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14390.565597] LNet: Removed LNI 192.168.201.158@tcp [14391.408576] Key type .llcrypt unregistered [14391.410776] Key type ._llcrypt unregistered [14407.749779] Key type ._llcrypt registered [14407.751869] Key type .llcrypt registered [14407.830437] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [14409.214497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14409.319448] alg: No test for adler32 (adler32-zlib) [14410.349436] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14410.664638] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [14412.335810] Key type lgssc registered [14413.620948] Lustre: Echo OBD driver; http://www.lustre.org/ [14417.871092] LNet: Added LNI 192.168.201.158@tcp100 [8/256/0/180] [14418.898404] LNet: Added LNI 192.168.201.158@tcp101 [8/256/0/180] [14419.625860] LNet: Added LNI 192.168.201.158@tcp102 [8/256/0/180] [14420.305845] LNet: Added LNI 192.168.201.158@tcp103 [8/256/0/180] [14421.017700] LNet: Added LNI 192.168.201.158@tcp104 [8/256/0/180] [14421.754498] LNet: Added LNI 192.168.201.158@tcp105 [8/256/0/180] [14422.595077] LNet: Added LNI 192.168.201.158@tcp106 [8/256/0/180] [14423.434650] LNet: Added LNI 192.168.201.158@tcp107 [8/256/0/180] [14424.164789] LNet: Added LNI 192.168.201.158@tcp108 [8/256/0/180] [14425.069670] LNet: Added LNI 192.168.201.158@tcp109 [8/256/0/180] [14426.195561] LNet: Added LNI 192.168.201.158@tcp110 [8/256/0/180] [14426.969562] LNet: Added LNI 192.168.201.158@tcp111 [8/256/0/180] [14427.645833] LNet: Added LNI 192.168.201.158@tcp112 [8/256/0/180] [14428.531575] LNet: Added LNI 192.168.201.158@tcp113 [8/256/0/180] [14429.628822] LNet: Added LNI 192.168.201.158@tcp114 [8/256/0/180] [14430.714854] LNet: Added LNI 192.168.201.158@tcp115 [8/256/0/180] [14431.649617] LNet: Added LNI 192.168.201.158@tcp116 [8/256/0/180] [14432.625792] LNet: Added LNI 192.168.201.158@tcp117 [8/256/0/180] [14433.545681] LNet: Added LNI 192.168.201.158@tcp118 [8/256/0/180] [14434.550254] LNet: Added LNI 192.168.201.158@tcp119 [8/256/0/180] [14435.524393] LNet: Added LNI 192.168.201.158@tcp120 [8/256/0/180] [14436.364890] LNet: Added LNI 192.168.201.158@tcp121 [8/256/0/180] [14437.200244] LNet: Added LNI 192.168.201.158@tcp122 [8/256/0/180] [14437.941925] LNet: Added LNI 192.168.201.158@tcp123 [8/256/0/180] [14438.736313] LNet: Added LNI 192.168.201.158@tcp124 [8/256/0/180] [14439.534874] LNet: Added LNI 192.168.201.158@tcp125 [8/256/0/180] [14440.281531] LNet: Added LNI 192.168.201.158@tcp126 [8/256/0/180] [14441.014444] LNet: Added LNI 192.168.201.158@tcp127 [8/256/0/180] [14441.940075] LNet: Added LNI 192.168.201.158@tcp128 [8/256/0/180] [14442.823069] LNet: Added LNI 192.168.201.158@tcp129 [8/256/0/180] [14443.743943] LNet: Added LNI 192.168.201.158@tcp130 [8/256/0/180] [14444.509187] LNet: Added LNI 192.168.201.158@tcp131 [8/256/0/180] [14445.350247] LNet: Added LNI 192.168.201.158@tcp132 [8/256/0/180] [14446.199745] LNet: Added LNI 192.168.201.158@tcp133 [8/256/0/180] [14447.162292] LNet: Added LNI 192.168.201.158@tcp134 [8/256/0/180] [14448.373574] LNet: Added LNI 192.168.201.158@tcp135 [8/256/0/180] [14455.155830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14455.185537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14456.314766] Lustre: 464522:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.158@tcp: can't add NID 192.168.201.158@tcp130: rc = -75 [14456.396893] LNetError: 464543:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.158 [14456.407521] LNetError: Protocol error connecting to 192.168.201.158@tcp118 at host 192.168.201.158:988: is it running a compatible version of Lustre? [14456.541474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14456.612715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14456.727325] Lustre: lustre-MDT0000: new disk, initializing [14456.735974] LNet: 3 local NIs in recovery (showing 3): 192.168.201.158@tcp123, 192.168.201.158@tcp129, 192.168.201.158@tcp128 [14456.797500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14456.824580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14457.439952] LNetError: 464550:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.158 [14457.459208] LNetError: 464550:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14457.462626] LNetError: Protocol error connecting to 192.168.201.158@tcp106 at host 192.168.201.158:988: is it running a compatible version of Lustre? [14457.468976] LNetError: Skipped 11 previous similar messages [14459.808711] LNetError: 464544:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.158 [14459.837228] LNetError: 464544:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [14459.850763] LNetError: Protocol error connecting to 192.168.201.158@tcp123 at host 192.168.201.158:988: is it running a compatible version of Lustre? [14459.872515] LNetError: Skipped 3 previous similar messages [14461.454379] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14463.907597] LNetError: 461983:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.158 [14463.922234] LNetError: Protocol error connecting to 192.168.201.158@tcp123 at host 192.168.201.158:988: is it running a compatible version of Lustre? [14471.072468] LNetError: 464546:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.158 [14471.081624] LNetError: Protocol error connecting to 192.168.201.158@tcp123 at host 192.168.201.158:988: is it running a compatible version of Lustre? [14472.244969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14472.429600] Lustre: 464541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14472.478458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14472.481491] Lustre: Skipped 1 previous similar message [14472.533827] Lustre: lustre-MDT0001: new disk, initializing [14472.568790] Lustre: 465341:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.158@tcp: can't add NID 192.168.201.158@tcp130: rc = -75 [14472.575310] Lustre: 465341:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14472.662559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14472.689122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14472.709830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14477.735818] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14482.462981] Lustre: 465765:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.158@tcp: can't add NID 192.168.201.158@tcp130: rc = -75 [14482.471492] Lustre: 465765:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14487.465300] LNetError: 464543:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.158 [14487.479629] LNetError: Protocol error connecting to 192.168.201.158@tcp123 at host 192.168.201.158:988: is it running a compatible version of Lustre? [14487.937826] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14493.951711] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14500.919245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14501.373331] Lustre: lustre-OST0000: new disk, initializing [14501.380479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14501.386635] Lustre: 466514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14501.486205] Lustre: 466519:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.158@tcp: can't add NID 192.168.201.158@tcp130: rc = -75 [14501.492384] Lustre: 466519:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14501.561378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14504.864971] LNetError: 464549:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.158 [14504.871199] LNetError: 464549:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [14504.889834] LNetError: Protocol error connecting to 192.168.201.158@tcp124 at host 192.168.201.158:988: is it running a compatible version of Lustre? [14504.906169] LNetError: Skipped 13 previous similar messages [14508.338683] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14511.119862] Lustre: 466970:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.158@tcp: can't add NID 192.168.201.158@tcp130: rc = -75 [14511.125656] Lustre: 466970:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14511.202861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14511.217403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14511.336955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14516.128944] LNet: 1 local NIs in recovery (showing 1): 192.168.201.158@tcp123 [14518.858619] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14526.945387] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14526.950873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14526.973843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14531.866683] Lustre: server umount lustre-OST0000 complete [14537.186404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14537.197614] Lustre: Skipped 1 previous similar message [14537.202388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14537.208654] Lustre: Skipped 2 previous similar messages [14541.817706] Lustre: server umount lustre-MDT0000 complete [14543.775864] LNetError: 464546:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.158 [14543.783032] LNetError: 464546:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 24 previous similar messages [14543.789888] LNetError: Protocol error connecting to 192.168.201.158@tcp124 at host 192.168.201.158:988: is it running a compatible version of Lustre? [14543.794287] LNetError: Skipped 24 previous similar messages [14545.296163] LustreError: 464532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454574 with bad export cookie 17066245861043234053 [14545.316587] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14545.319549] LustreError: 464532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14545.750665] Lustre: server umount lustre-MDT0001 complete [14558.218407] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [14561.606408] Key type lgssc unregistered [14562.057276] LNet: 468433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14562.075426] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14562.082724] LNet: Removed LNI 192.168.201.158@tcp [14563.103259] LNet: Removed LNI 192.168.201.158@tcp100 [14563.104808] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14563.114333] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14563.122424] LNet: Removed LNI 192.168.201.158@tcp101 [14564.191653] LNet: Removed LNI 192.168.201.158@tcp102 [14564.207130] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14564.217331] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14564.234892] LNet: Removed LNI 192.168.201.158@tcp103 [14565.283323] LNet: Removed LNI 192.168.201.158@tcp104 [14566.303628] LNet: Removed LNI 192.168.201.158@tcp105 [14566.305423] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14566.310761] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14566.317697] LNet: Removed LNI 192.168.201.158@tcp106 [14567.396043] LNet: Removed LNI 192.168.201.158@tcp107 [14568.479892] LNet: Removed LNI 192.168.201.158@tcp108 [14569.503190] LNet: Removed LNI 192.168.201.158@tcp109 [14570.527238] LNet: Removed LNI 192.168.201.158@tcp110 [14570.539704] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14570.544738] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14571.551581] LNet: Removed LNI 192.168.201.158@tcp111 [14572.575320] LNet: Removed LNI 192.168.201.158@tcp112 [14573.599429] LNet: Removed LNI 192.168.201.158@tcp113 [14574.623961] LNet: Removed LNI 192.168.201.158@tcp114 [14575.649518] LNet: Removed LNI 192.168.201.158@tcp115 [14576.671194] LNet: Removed LNI 192.168.201.158@tcp116 [14577.703698] LNet: Removed LNI 192.168.201.158@tcp117 [14578.721267] LNet: Removed LNI 192.168.201.158@tcp118 [14578.727314] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14578.734732] LNetError: 468433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [14578.758278] LNet: Removed LNI 192.168.201.158@tcp119 [14579.807732] LNet: Removed LNI 192.168.201.158@tcp120 [14580.831562] LNet: Removed LNI 192.168.201.158@tcp121 [14581.855796] LNet: Removed LNI 192.168.201.158@tcp122 [14581.859148] LNet: Removed LNI 192.168.201.158@tcp123 [14582.879164] LNet: Removed LNI 192.168.201.158@tcp124 [14583.905332] LNet: Removed LNI 192.168.201.158@tcp125 [14584.927166] LNet: Removed LNI 192.168.201.158@tcp126 [14585.956521] LNet: Removed LNI 192.168.201.158@tcp127 [14586.975236] LNet: Removed LNI 192.168.201.158@tcp128 [14587.999394] LNet: Removed LNI 192.168.201.158@tcp129 [14588.002977] LNet: Removed LNI 192.168.201.158@tcp130 [14588.004556] LNet: Removed LNI 192.168.201.158@tcp131 [14588.008432] LNet: Removed LNI 192.168.201.158@tcp132 [14588.019163] LNet: Removed LNI 192.168.201.158@tcp133 [14588.023332] LNet: Removed LNI 192.168.201.158@tcp134 [14588.036040] LNet: Removed LNI 192.168.201.158@tcp135 [14588.869788] Key type .llcrypt unregistered [14588.872900] Key type ._llcrypt unregistered [14612.431577] Key type ._llcrypt registered [14612.436449] Key type .llcrypt registered [14612.562220] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [14627.428577] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [14628.179228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14628.201224] alg: No test for adler32 (adler32-zlib) [14629.462433] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14629.814118] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [14631.535948] Key type lgssc registered [14632.645728] Lustre: Echo OBD driver; http://www.lustre.org/ [14667.829691] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:58:15 (1788454695) [14685.398240] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [14687.730964] Key type lgssc unregistered [14688.082690] LNet: 472526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14688.094769] LNetError: 472526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14688.104821] LNet: Removed LNI 192.168.201.158@tcp [14688.946566] Key type .llcrypt unregistered [14688.948423] Key type ._llcrypt unregistered [14705.308912] Key type ._llcrypt registered [14705.310874] Key type .llcrypt registered [14705.412320] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [14706.502893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14706.645227] alg: No test for adler32 (adler32-zlib) [14707.680381] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14707.896895] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [14709.575187] Key type lgssc registered [14710.459347] Lustre: Echo OBD driver; http://www.lustre.org/ [14719.492117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14719.531354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14720.747552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14720.770633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14720.827979] Lustre: lustre-MDT0000: new disk, initializing [14720.893852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14720.906551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14725.203725] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14736.273677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14736.463866] Lustre: 473737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14736.523167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14736.526643] Lustre: Skipped 1 previous similar message [14736.626260] Lustre: lustre-MDT0001: new disk, initializing [14736.736190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14736.802553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14736.814445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14740.885893] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14750.634485] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14757.387626] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14765.541570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14765.861536] Lustre: lustre-OST0000: new disk, initializing [14765.867384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14765.881640] Lustre: 475708:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14765.977407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14773.544499] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [14775.921583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14775.940971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14776.067056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14783.443895] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14787.483938] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing lnet_if_list [14792.733954] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14792.770808] LNet: Added LNI 192.168.201.158@tcp10 [8/256/0/180] [14798.686414] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14798.700374] LNet: Added LNI 192.168.201.158@tcp11 [8/256/0/180] [14802.121157] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14802.138911] LNet: Added LNI 192.168.201.158@tcp12 [8/256/0/180] [14806.446568] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14806.509060] LNet: Added LNI 192.168.201.158@tcp13 [8/256/0/180] [14811.508033] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14811.535646] LNet: Added LNI 192.168.201.158@tcp14 [8/256/0/180] [14816.801168] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14816.847614] LNet: Added LNI 192.168.201.158@tcp15 [8/256/0/180] [14821.380439] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14821.398526] LNet: Added LNI 192.168.201.158@tcp16 [8/256/0/180] [14826.245812] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14826.279226] LNet: Added LNI 192.168.201.158@tcp17 [8/256/0/180] [14830.894840] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14830.926890] LNet: Added LNI 192.168.201.158@tcp18 [8/256/0/180] [14836.407883] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14836.449299] LNet: Added LNI 192.168.201.158@tcp19 [8/256/0/180] [14842.098736] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14842.141971] LNet: Added LNI 192.168.201.158@tcp20 [8/256/0/180] [14847.957143] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14848.000561] LNet: Added LNI 192.168.201.158@tcp21 [8/256/0/180] [14853.575022] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14853.642992] LNet: Added LNI 192.168.201.158@tcp22 [8/256/0/180] [14859.618140] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14859.647158] LNet: Added LNI 192.168.201.158@tcp23 [8/256/0/180] [14865.750157] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14865.817803] LNet: Added LNI 192.168.201.158@tcp24 [8/256/0/180] [14871.153852] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14871.192898] LNet: Added LNI 192.168.201.158@tcp25 [8/256/0/180] [14877.818712] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14877.867969] LNet: Added LNI 192.168.201.158@tcp26 [8/256/0/180] [14885.449458] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14885.530692] LNet: Added LNI 192.168.201.158@tcp27 [8/256/0/180] [14895.305419] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14895.343182] LNet: Added LNI 192.168.201.158@tcp28 [8/256/0/180] [14903.497668] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14903.550574] LNet: Added LNI 192.168.201.158@tcp29 [8/256/0/180] [14916.280533] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14916.374780] LNet: Added LNI 192.168.201.158@tcp30 [8/256/0/180] [14925.556839] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14925.618980] LNet: Added LNI 192.168.201.158@tcp31 [8/256/0/180] [14933.276335] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14933.317480] LNet: Added LNI 192.168.201.158@tcp32 [8/256/0/180] [14942.744019] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14942.794066] LNet: Added LNI 192.168.201.158@tcp33 [8/256/0/180] [14949.627910] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14949.658942] LNet: Added LNI 192.168.201.158@tcp34 [8/256/0/180] [14957.644928] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14957.669796] LNet: Added LNI 192.168.201.158@tcp35 [8/256/0/180] [14965.591855] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14965.639274] LNet: Added LNI 192.168.201.158@tcp36 [8/256/0/180] [14975.210934] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14975.257666] LNet: Added LNI 192.168.201.158@tcp37 [8/256/0/180] [14983.501033] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14983.527172] LNet: Added LNI 192.168.201.158@tcp38 [8/256/0/180] [14995.188429] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14995.215400] LNet: Added LNI 192.168.201.158@tcp39 [8/256/0/180] [15005.676869] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15005.741174] LNet: Added LNI 192.168.201.158@tcp40 [8/256/0/180] [15013.375263] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15013.428902] LNet: Added LNI 192.168.201.158@tcp41 [8/256/0/180] [15022.564710] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15022.623480] LNet: Added LNI 192.168.201.158@tcp42 [8/256/0/180] [15031.993263] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15032.034904] LNet: Added LNI 192.168.201.158@tcp43 [8/256/0/180] [15041.967941] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15042.024326] LNet: Added LNI 192.168.201.158@tcp44 [8/256/0/180] [15052.167980] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15052.205268] LNet: Added LNI 192.168.201.158@tcp45 [8/256/0/180] [15061.158382] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15061.211338] LNet: Added LNI 192.168.201.158@tcp46 [8/256/0/180] [15072.947958] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15072.968596] LNet: Added LNI 192.168.201.158@tcp47 [8/256/0/180] [15083.633396] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15083.681991] LNet: Added LNI 192.168.201.158@tcp48 [8/256/0/180] [15094.802555] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15094.854615] LNet: Added LNI 192.168.201.158@tcp49 [8/256/0/180] [15106.382293] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15106.413453] LNet: Added LNI 192.168.201.158@tcp50 [8/256/0/180] [15131.104828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15131.138431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15131.164807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15134.688536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15134.690814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15134.701488] Lustre: Skipped 1 previous similar message [15134.713608] Lustre: Skipped 2 previous similar messages [15137.300956] Lustre: server umount lustre-MDT0000 complete [15141.921237] LustreError: 473729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455171 with bad export cookie 3754525348453230513 [15141.929501] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15141.939145] LustreError: 473729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15142.575711] Lustre: server umount lustre-MDT0001 complete [15161.125072] Lustre: 473010:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455174/real 1788455174] req@00000000a1040940 x1875330713441152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455190 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15161.162805] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15163.359178] Lustre: 473009:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455176/real 1788455176] req@00000000a4fa97c2 x1875330713441280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15165.069782] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [15166.436220] Lustre: 473008:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455179/real 1788455179] req@00000000727c9271 x1875330713441408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15168.486474] Lustre: 473011:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455181/real 1788455181] req@00000000942e6f78 x1875330713441664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15173.666313] Lustre: 473011:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455186/real 1788455186] req@0000000023878725 x1875330713441792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15174.727106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15187.810444] LustreError: 473007:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000010a2ea75 x1875330713442560/t0(0) o250->MGC192.168.201.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15188.209121] LustreError: 475700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15188.248173] LustreError: 475700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15188.321937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15190.375459] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15190.389984] LustreError: 475701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15191.392735] LustreError: 475700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15193.624935] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15196.658112] LustreError: 475701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15201.774497] LustreError: 475700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15204.003245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15204.630977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15206.704504] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15209.835343] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15220.525653] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15227.637394] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15247.843917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15247.863350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15247.877521] Lustre: Skipped 1 previous similar message [15247.894822] Lustre: Skipped 1 previous similar message [15252.966914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15252.970431] Lustre: Skipped 1 previous similar message [15253.773215] Lustre: server umount lustre-OST0000 complete [15258.081765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15258.089548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15258.095666] Lustre: Skipped 1 previous similar message [15258.105399] Lustre: Skipped 1 previous similar message [15263.745193] Lustre: server umount lustre-MDT0000 complete [15267.982970] LustreError: 473730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455297 with bad export cookie 3754525348453233096 [15267.991687] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15267.999326] LustreError: 473730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15269.011422] Lustre: server umount lustre-MDT0001 complete [15286.210967] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [15290.132552] Key type lgssc unregistered [15290.791783] LNet: 502796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15290.804986] LNetError: 502796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15290.821185] LNet: Removed LNI 192.168.201.158@tcp [15290.825100] LNet: Removed LNI 192.168.201.158@tcp10 [15290.836692] LNet: Removed LNI 192.168.201.158@tcp11 [15290.846501] LNet: Removed LNI 192.168.201.158@tcp12 [15290.856123] LNet: Removed LNI 192.168.201.158@tcp13 [15290.861486] LNet: Removed LNI 192.168.201.158@tcp14 [15290.868347] LNet: Removed LNI 192.168.201.158@tcp15 [15290.873726] LNet: Removed LNI 192.168.201.158@tcp16 [15290.875648] LNet: Removed LNI 192.168.201.158@tcp17 [15290.878852] LNet: Removed LNI 192.168.201.158@tcp18 [15290.886019] LNet: Removed LNI 192.168.201.158@tcp19 [15290.889122] LNet: Removed LNI 192.168.201.158@tcp20 [15290.896057] LNet: Removed LNI 192.168.201.158@tcp21 [15290.902615] LNet: Removed LNI 192.168.201.158@tcp22 [15290.904446] LNet: Removed LNI 192.168.201.158@tcp23 [15290.911670] LNet: Removed LNI 192.168.201.158@tcp24 [15290.916101] LNet: Removed LNI 192.168.201.158@tcp25 [15290.926886] LNet: Removed LNI 192.168.201.158@tcp26 [15290.937032] LNet: Removed LNI 192.168.201.158@tcp27 [15290.940311] LNet: Removed LNI 192.168.201.158@tcp28 [15290.947956] LNet: Removed LNI 192.168.201.158@tcp29 [15290.949949] LNet: Removed LNI 192.168.201.158@tcp30 [15290.956155] LNet: Removed LNI 192.168.201.158@tcp31 [15290.963167] LNet: Removed LNI 192.168.201.158@tcp32 [15290.967941] LNet: Removed LNI 192.168.201.158@tcp33 [15290.974357] LNet: Removed LNI 192.168.201.158@tcp34 [15290.978428] LNet: Removed LNI 192.168.201.158@tcp35 [15290.985944] LNet: Removed LNI 192.168.201.158@tcp36 [15290.989681] LNet: Removed LNI 192.168.201.158@tcp37 [15290.993670] LNet: Removed LNI 192.168.201.158@tcp38 [15290.995375] LNet: Removed LNI 192.168.201.158@tcp39 [15291.002888] LNet: Removed LNI 192.168.201.158@tcp40 [15291.009767] LNet: Removed LNI 192.168.201.158@tcp41 [15291.012832] LNet: Removed LNI 192.168.201.158@tcp42 [15291.015750] LNet: Removed LNI 192.168.201.158@tcp43 [15291.019146] LNet: Removed LNI 192.168.201.158@tcp44 [15291.022983] LNet: Removed LNI 192.168.201.158@tcp45 [15291.026089] LNet: Removed LNI 192.168.201.158@tcp46 [15291.029993] LNet: Removed LNI 192.168.201.158@tcp47 [15291.042256] LNet: Removed LNI 192.168.201.158@tcp48 [15291.044483] LNet: Removed LNI 192.168.201.158@tcp49 [15291.061043] LNet: Removed LNI 192.168.201.158@tcp50 [15292.120974] Key type .llcrypt unregistered [15292.123219] Key type ._llcrypt unregistered [15316.176260] Key type ._llcrypt registered [15316.178724] Key type .llcrypt registered [15316.304644] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [15333.460665] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [15334.741202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15334.830025] alg: No test for adler32 (adler32-zlib) [15335.979324] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15336.404556] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [15338.128073] Key type lgssc registered [15339.644582] Lustre: Echo OBD driver; http://www.lustre.org/ [15377.737519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15377.788352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15379.243917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15379.310221] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15379.396345] Lustre: lustre-MDT0000: new disk, initializing [15379.557528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15379.594258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15385.339777] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15401.106279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15401.297970] Lustre: 505996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15401.389793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15401.406774] Lustre: Skipped 1 previous similar message [15401.581134] Lustre: lustre-MDT0001: new disk, initializing [15401.834283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15401.918922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15401.963807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15409.377581] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15420.445845] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15429.293708] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15439.569986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15439.774710] Lustre: *** cfs_fail_loc=90f, val=0*** [15439.849369] Lustre: lustre-OST0000: new disk, initializing [15439.853907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15439.859292] Lustre: 508015:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15439.954704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15445.613405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15445.621541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15445.718566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15446.779389] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15458.937885] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15471.587474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15471.608679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15471.620293] Lustre: Skipped 1 previous similar message [15471.644686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15471.656918] Lustre: Skipped 1 previous similar message [15476.705194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15481.839690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15481.847426] Lustre: Skipped 2 previous similar messages [15482.335145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15482.825666] Lustre: server umount lustre-OST0000 complete [15486.948481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15486.951190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15486.962134] Lustre: Skipped 1 previous similar message [15486.977421] Lustre: Skipped 1 previous similar message [15492.065169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15492.074444] Lustre: Skipped 1 previous similar message [15493.195745] Lustre: server umount lustre-MDT0000 complete [15496.619565] LustreError: 505990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455525 with bad export cookie 6302453540963452089 [15496.624315] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15496.628519] LustreError: 505990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15497.105504] Lustre: server umount lustre-MDT0001 complete [15507.679622] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [15512.285547] Key type lgssc unregistered [15512.738352] LNet: 509929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15512.749600] LNetError: 509929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15512.788823] LNet: Removed LNI 192.168.201.158@tcp [15514.024334] Key type .llcrypt unregistered [15514.027190] Key type ._llcrypt unregistered [15539.729652] Key type ._llcrypt registered [15539.732556] Key type .llcrypt registered [15539.935477] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [15541.644496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15541.741519] alg: No test for adler32 (adler32-zlib) [15543.259812] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15543.723861] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [15545.431185] Key type lgssc registered [15547.285694] Lustre: Echo OBD driver; http://www.lustre.org/ [15556.897618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15556.933262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15558.526170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15563.099868] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15568.867921] LustreError: 511202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15568.889603] LustreError: 511202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15571.908153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15572.394543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15577.439783] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15584.768610] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15591.392411] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15599.988411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15600.447243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15609.220487] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15618.848486] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15630.228701] loop0: detected capacity change from 0 to 204800000 [15630.508986] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15630.538925] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15630.546367] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15635.999610] loop0: detected capacity change from 0 to 204800000 [15636.220649] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15636.228702] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15636.243688] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15641.719479] loop0: detected capacity change from 0 to 204800000 [15646.682800] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15646.782715] Lustre: fs15246-OST0000: new disk, initializing [15646.786139] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15646.790624] Lustre: 514797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15646.838446] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15652.800349] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15662.361165] loop1: detected capacity change from 0 to 204800000 [15668.816501] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15668.959948] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15669.053202] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15669.194198] Lustre: fs15246-MDT0000: new disk, initializing [15669.509183] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15669.548280] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15669.711460] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15670.754677] Lustre: 515922:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000008632344a x1875331589912704/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788455715 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15674.127848] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15679.851063] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15679.867914] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15679.891199] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15697.403829] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15697.420355] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15701.658699] Lustre: server umount fs15246-MDT0000 complete [15702.501371] LustreError: 511198:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15707.622859] LustreError: 511198:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15712.453671] Lustre: server umount fs15246-OST0000 complete [15726.573558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15726.583748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15726.617047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15728.097487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15728.102141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15733.237086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15733.250952] Lustre: Skipped 2 previous similar messages [15738.339302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15738.349351] Lustre: Skipped 1 previous similar message [15739.871210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15740.078070] Lustre: server umount lustre-OST0000 complete [15746.529306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15746.544155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15746.579112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15750.730898] Lustre: server umount lustre-MDT0000 complete [15753.700664] LustreError: 513042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15753.736025] LustreError: 513042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15755.652109] LustreError: 513046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455784 with bad export cookie 10722961516591849706 [15755.661736] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15755.664973] LustreError: 513046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15756.067967] Lustre: server umount lustre-MDT0001 complete [15769.345423] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [15772.814081] Key type lgssc unregistered [15773.156781] LNet: 519023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15773.167733] LNetError: 519023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15773.196234] LNet: Removed LNI 192.168.201.158@tcp [15774.302155] Key type .llcrypt unregistered [15774.312463] Key type ._llcrypt unregistered [15794.303185] Key type ._llcrypt registered [15794.305265] Key type .llcrypt registered [15794.391765] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [15795.577076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15795.662890] alg: No test for adler32 (adler32-zlib) [15796.735289] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15796.910849] LNet: Added LNI 192.168.201.158@tcp [8/256/0/180] [15798.535703] Key type lgssc registered [15799.556333] Lustre: Echo OBD driver; http://www.lustre.org/ [15841.314487] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [15852.129919] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [15888.941996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15888.961358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15890.200202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15890.232467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15890.346095] Lustre: lustre-MDT0000: new disk, initializing [15890.464761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15890.496882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15895.349701] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15907.675635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15907.897349] Lustre: 523492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15908.007225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15908.016225] Lustre: Skipped 1 previous similar message [15908.144559] Lustre: lustre-MDT0001: new disk, initializing [15908.294130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15908.378449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15908.393801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15912.943194] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15924.129554] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15930.265295] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15937.391559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15937.754754] Lustre: lustre-OST0000: new disk, initializing [15937.762445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15937.773161] Lustre: 525459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15937.909585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15944.687182] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_default_debug -1 all [15947.392446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15947.409207] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15947.533082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15955.092977] Lustre: DEBUG MARKER: oleg158-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15961.690671] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15962.028696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15968.166687] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15968.367246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15973.360754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15973.369906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15973.388167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15977.447579] Lustre: server umount lustre-OST0000 complete [15983.588032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15983.601756] Lustre: Skipped 2 previous similar messages [15983.602559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15983.619290] Lustre: Skipped 2 previous similar messages [15986.974779] Lustre: server umount lustre-MDT0000 complete [15990.221792] LustreError: 523482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456019 with bad export cookie 10029701735605139864 [15990.230709] LustreError: MGC192.168.201.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15990.233923] LustreError: 523482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15990.616545] Lustre: server umount lustre-MDT0001 complete [16019.121643] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing set_hostid [16034.411983] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing load_modules_local [16068.217633] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15756 sec ========= 13:21:35 (1788456095) [16070.053552] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:21:37 (1788456097) === [16073.676764] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:21:41 (1788456101) === [16113.066335] Lustre: DEBUG MARKER: oleg158-server.virtnet: executing unload_modules_local [16116.168536] Key type lgssc unregistered [16116.490687] LNet: 532142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16116.496633] LNetError: 532142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16117.543864] LNet: Removed LNI 192.168.201.158@tcp [16118.644574] Key type .llcrypt unregistered [16118.646746] Key type ._llcrypt unregistered