[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495685976 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002366] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006980] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009106] LSM: Security Framework initializing [ 0.010048] Yama: becoming mindful. [ 0.011034] SELinux: Initializing. [ 0.012058] *** VALIDATE selinux *** [ 0.020276] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024791] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027083] *** VALIDATE tmpfs *** [ 0.028357] *** VALIDATE proc *** [ 0.029209] *** VALIDATE cgroup *** [ 0.030010] *** VALIDATE cgroup2 *** [ 0.031246] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034027] Spectre V2 : User space: Vulnerable [ 0.035009] Speculative Store Bypass: Vulnerable [ 0.037525] debug: unmapping init [mem 0xffffffff84059000-0xffffffff84060fff] [ 0.039155] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040618] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041019] ... version: 2 [ 0.042021] ... bit width: 48 [ 0.043011] ... generic registers: 4 [ 0.044024] ... value mask: 0000ffffffffffff [ 0.045020] ... max period: 00007fffffffffff [ 0.046017] ... fixed-purpose events: 3 [ 0.047014] ... event mask: 000000070000000f [ 0.048314] rcu: Hierarchical SRCU implementation. [ 0.050414] smp: Bringing up secondary CPUs ... [ 0.051629] x86: Booting SMP configuration: [ 0.052026] .... node #0, CPUs: #1 #2 #3 [ 0.057088] smp: Brought up 1 node, 4 CPUs [ 0.059015] smpboot: Max logical packages: 1 [ 0.060012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136127] node 0 deferred pages initialised in 73ms [ 0.139461] devtmpfs: initialized [ 0.140232] x86/mm: Memory block size: 128MB [ 0.142794] gcov: version magic: 0x41383552 [ 0.146082] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152265] pinctrl core: initialized pinctrl subsystem [ 0.154149] [ 0.154701] ************************************************************* [ 0.157013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160012] ** ** [ 0.162060] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165013] ** ** [ 0.167012] ** This means that this kernel is built to expose internal ** [ 0.170018] ** IOMMU data structures, which may compromise security on ** [ 0.174012] ** your system. ** [ 0.177011] ** ** [ 0.179012] ** If you see this message and you are not debugging the ** [ 0.182015] ** kernel, report this immediately to your vendor! ** [ 0.185012] ** ** [ 0.188015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191140] ************************************************************* [ 0.195623] NET: Registered protocol family 16 [ 0.196442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201105] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.205408] cpuidle: using governor menu [ 0.206869] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208426] PCI: Using configuration type 1 for base access [ 0.210123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.221075] cryptd: max_cpu_qlen set to 1000 [ 0.224270] ACPI: Added _OSI(Module Device) [ 0.225000] ACPI: Added _OSI(Processor Device) [ 0.226016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.228014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233069] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240449] ACPI: Interpreter enabled [ 0.242056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243010] ACPI: Using IOAPIC for interrupt routing [ 0.245104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.248373] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258302] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.264016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.272416] acpiphp: Slot [2] registered [ 0.275009] acpiphp: Slot [5] registered [ 0.276153] acpiphp: Slot [6] registered [ 0.278097] acpiphp: Slot [7] registered [ 0.279118] acpiphp: Slot [8] registered [ 0.281110] acpiphp: Slot [9] registered [ 0.282153] acpiphp: Slot [10] registered [ 0.284127] acpiphp: Slot [3] registered [ 0.285124] acpiphp: Slot [4] registered [ 0.287071] acpiphp: Slot [11] registered [ 0.288085] acpiphp: Slot [12] registered [ 0.290113] acpiphp: Slot [13] registered [ 0.291102] acpiphp: Slot [14] registered [ 0.293098] acpiphp: Slot [15] registered [ 0.294095] acpiphp: Slot [16] registered [ 0.295092] acpiphp: Slot [17] registered [ 0.297096] acpiphp: Slot [18] registered [ 0.299109] acpiphp: Slot [19] registered [ 0.300101] acpiphp: Slot [20] registered [ 0.302119] acpiphp: Slot [21] registered [ 0.304100] acpiphp: Slot [22] registered [ 0.306104] acpiphp: Slot [23] registered [ 0.308102] acpiphp: Slot [24] registered [ 0.309112] acpiphp: Slot [25] registered [ 0.310256] acpiphp: Slot [26] registered [ 0.312104] acpiphp: Slot [27] registered [ 0.314093] acpiphp: Slot [28] registered [ 0.315079] acpiphp: Slot [29] registered [ 0.316084] acpiphp: Slot [30] registered [ 0.318091] acpiphp: Slot [31] registered [ 0.319057] PCI host bridge to bus 0000:00 [ 0.321013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.323020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.325020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.333027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336130] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.338958] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.343367] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352784] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.359624] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.369776] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.371684] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.376682] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.383014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.397014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.404014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409763] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.425016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.432013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.450015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.461012] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.468015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.475073] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.494018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.505243] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.513020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.521019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.537021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.549118] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.558019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.567018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.585020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.596708] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.603014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.609019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.630019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.643881] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.651019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.660020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.676025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.687937] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.692441] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.694562] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.697557] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.701313] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.706146] iommu: Default domain type: Passthrough [ 0.710058] SCSI subsystem initialized [ 0.711461] ACPI: bus type USB registered [ 0.714210] usbcore: registered new interface driver usbfs [ 0.717110] usbcore: registered new interface driver hub [ 0.719148] usbcore: registered new device driver usb [ 0.721200] pps_core: LinuxPPS API ver. 1 registered [ 0.723015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.727085] PTP clock support registered [ 0.729105] EDAC MC: Ver: 3.0.0 [ 0.730273] PCI: Using ACPI for IRQ routing [ 0.733121] NetLabel: Initializing [ 0.734013] NetLabel: domain hash size = 128 [ 0.736012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.737093] NetLabel: unlabeled traffic allowed by default [ 0.740050] vgaarb: loaded [ 0.741349] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.750000] clocksource: Switched to clocksource kvm-clock [ 0.862882] VFS: Disk quotas dquot_6.6.0 [ 0.865138] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.868213] *** VALIDATE ramfs *** [ 0.869821] *** VALIDATE hugetlbfs *** [ 0.871464] pnp: PnP ACPI init [ 0.873935] pnp: PnP ACPI: found 6 devices [ 0.890359] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.895942] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.898237] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.900786] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.903492] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.906137] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909253] NET: Registered protocol family 2 [ 0.911892] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.917145] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921068] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.926601] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.929914] TCP: Hash tables configured (established 65536 bind 65536) [ 0.932958] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.936446] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941047] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944397] NET: Registered protocol family 1 [ 0.947192] RPC: Registered named UNIX socket transport module. [ 0.950050] RPC: Registered udp transport module. [ 0.952330] RPC: Registered tcp transport module. [ 0.954646] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.957403] NET: Registered protocol family 44 [ 0.959202] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.961669] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.965099] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.967543] PCI: CLS 0 bytes, default 64 [ 0.969374] Unpacking initramfs... [ 2.419864] debug: unmapping init [mem 0xffff9e9c7cc54000-0xffff9e9c7ffbffff] [ 2.426992] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.429846] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.433737] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.941702] Initialise system trusted keyrings [ 2.943830] Key type blacklist registered [ 2.945751] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.960400] zbud: loaded [ 2.964916] *** VALIDATE nfs *** [ 2.966185] *** VALIDATE nfs4 *** [ 2.967783] pstore: using deflate compression [ 2.970929] Platform Keyring initialized [ 3.089581] NET: Registered protocol family 38 [ 3.091417] Key type asymmetric registered [ 3.092906] Asymmetric key parser 'x509' registered [ 3.094849] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.098728] io scheduler mq-deadline registered [ 3.100309] io scheduler kyber registered [ 3.102177] io scheduler bfq registered [ 3.104153] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.107786] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.112373] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.115233] ACPI: Power Button [PWRF] [ 3.123128] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.131368] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.147087] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.155071] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.175819] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.205280] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.233540] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.239579] Non-volatile memory driver v1.3 [ 3.241039] Linux agpgart interface v0.103 [ 3.269854] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.272963] vda: detected capacity change from 0 to 74829824 [ 3.298124] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.302696] vdb: detected capacity change from 0 to 1073741824 [ 3.321167] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.324680] vdc: detected capacity change from 0 to 2621440000 [ 3.340653] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.343796] vdd: detected capacity change from 0 to 2621440000 [ 3.361719] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.365663] vde: detected capacity change from 0 to 4294967296 [ 3.381233] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.385197] vdf: detected capacity change from 0 to 4294967296 [ 3.395672] libphy: Fixed MDIO Bus: probed [ 3.404274] usbcore: registered new interface driver usbserial_generic [ 3.410582] usbserial: USB Serial support registered for generic [ 3.414282] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.420584] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.422546] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.425813] mousedev: PS/2 mouse device common for all mice [ 3.428642] rtc_cmos 00:05: RTC can wake from S4 [ 3.431288] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.432836] rtc_cmos 00:05: registered as rtc0 [ 3.436977] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.438111] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.443401] intel_pstate: CPU model not supported [ 3.444276] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.449548] hid: raw HID events driver (C) Jiri Kosina [ 3.451327] usbcore: registered new interface driver usbhid [ 3.453059] usbhid: USB HID core driver [ 3.454584] drop_monitor: Initializing network drop monitor service [ 3.456636] Initializing XFRM netlink socket [ 3.458399] NET: Registered protocol family 10 [ 3.461243] Segment Routing with IPv6 [ 3.462351] NET: Registered protocol family 17 [ 3.464722] mpls_gso: MPLS GSO support [ 3.471645] RAS: Correctable Errors collector initialized. [ 3.474278] AVX version of gcm_enc/dec engaged. [ 3.476422] AES CTR mode by8 optimization enabled [ 3.555380] sched_clock: Marking stable (3555347343, 0)->(4440636531, -885289188) [ 3.559288] registered taskstats version 1 [ 3.561578] Loading compiled-in X.509 certificates [ 3.563703] zswap: loaded using pool lzo/zbud [ 3.589299] Key type big_key registered [ 3.600480] Key type encrypted registered [ 3.602155] ima: No TPM chip found, activating TPM-bypass! [ 3.604314] ima: Allocated hash algorithm: sha1 [ 3.605660] ima: No architecture policies found [ 3.607413] evm: Initialising EVM extended attributes: [ 3.609620] evm: security.selinux [ 3.610554] evm: security.ima [ 3.611349] evm: security.capability [ 3.612885] evm: HMAC attrs: 0x1 [ 3.615329] rtc_cmos 00:05: setting system clock to 2026-08-24 12:19:44 UTC (1787573984) [ 3.621530] debug: unmapping init [mem 0xffffffff85003000-0xffffffff851fffff] [ 3.624659] debug: unmapping init [mem 0xffffffff83d82000-0xffffffff84058fff] [ 3.636163] Write protecting the kernel read-only data: 28672k [ 3.639811] debug: unmapping init [mem 0xffffffff82403000-0xffffffff825fffff] [ 3.642618] debug: unmapping init [mem 0xffffffff82d14000-0xffffffff82dfffff] [ 3.680970] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.687819] systemd[1]: Detected virtualization kvm. [ 3.690163] systemd[1]: Detected architecture x86-64. [ 3.692113] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.715327] systemd[1]: No hostname configured. [ 3.717079] systemd[1]: Set hostname to . [ 3.720463] random: systemd: uninitialized urandom read (16 bytes read) [ 3.723151] systemd[1]: Initializing machine ID from random generator. [ 3.770838] random: ln: uninitialized urandom read (6 bytes read) [ 3.864920] random: systemd: uninitialized urandom read (16 bytes read) [ 3.868733] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.878470] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 3.890735] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.527787] device-mapper: uevent: version 1.0.3 [ 4.530583] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.382670] virtio_net virtio0 ens2: renamed from eth0 [ 5.400737] random: fast init done [ 5.457353] scsi host0: ata_piix [ 5.486816] scsi host1: ata_piix [ 5.489442] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.492282] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.322852] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.117804] random: crng init done [ 10.118984] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.738455] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.929968] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.213781] SELinux: Disabled at runtime. [ 12.277635] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.286817] systemd[1]: Detected virtualization kvm. [ 12.288378] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.856718] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.861094] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.868610] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.872468] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.875120] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.882886] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.891299] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Consol[ 13.042358] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.419453] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.734652] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.834375] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.896197] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.905636] EDAC sbridge: Ver: 1.1.2 [ 15.780612] Key type dns_resolver registered [ 16.116784] NFS: Registering the id_resolver key type [ 16.119608] Key type id_resolver registered [ 16.121551] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg640-server login: [ 42.041256] hrtimer: interrupt took 2322496 ns [ 68.851906] libcfs: loading out-of-tree module taints kernel. [ 68.929694] Key type ._llcrypt registered [ 68.934688] Key type .llcrypt registered [ 69.106899] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 86.739111] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 88.236967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 88.250206] alg: No test for adler32 (adler32-zlib) [ 89.702531] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 90.437761] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 92.167329] Key type lgssc registered [ 93.934918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.698853] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 152.187495] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 168.301393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.368090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.748039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.797068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 169.938919] Lustre: lustre-MDT0000: new disk, initializing [ 170.070798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.105640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 174.640483] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.964834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 187.051985] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.074816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 187.078639] Lustre: Skipped 1 previous similar message [ 187.188703] Lustre: lustre-MDT0001: new disk, initializing [ 187.253208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.275626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.284097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.066651] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.766816] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.816320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.085649] Lustre: lustre-OST0000: new disk, initializing [ 205.091666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.100169] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.161116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.266839] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.537219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 213.548594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 213.614385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 224.943489] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.094259] Lustre: lustre-OST0001: new disk, initializing [ 225.098245] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 225.105618] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.185653] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.358903] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.485557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 233.493798] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 233.556121] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.223718] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.425390] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 253.470924] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing check_logdir /tmp/testlogs/ [ 260.385729] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing yml_node [ 265.940921] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 268.987878] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 272.589263] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 274.819630] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 08:24:13 EDT 2026 [ 295.705528] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 297.721760] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 315.364622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 315.365052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.401384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.406520] Lustre: Skipped 3 previous similar messages [ 320.480210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.483536] Lustre: Skipped 3 previous similar messages [ 320.628882] Lustre: server umount lustre-MDT0000 complete [ 324.409860] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574305 with bad export cookie 16075726531865699943 [ 324.411942] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.414786] LustreError: 6503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 325.606521] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.618968] Lustre: Skipped 4 previous similar messages [ 325.619224] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.631515] Lustre: Skipped 1 previous similar message [ 329.633378] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.648579] Lustre: Skipped 1 previous similar message [ 330.758219] Lustre: server umount lustre-MDT0001 complete [ 334.717405] Lustre: server umount lustre-OST0000 complete [ 338.100878] Lustre: server umount lustre-OST0001 complete [ 344.087298] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 353.882256] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 389.596936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.804148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 389.832466] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 389.932104] Lustre: lustre-MDT0000: new disk, initializing [ 390.015331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.044699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 393.507649] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 403.652415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.760948] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 403.811258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 403.815407] Lustre: Skipped 1 previous similar message [ 403.925930] Lustre: lustre-MDT0001: new disk, initializing [ 403.990325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.020352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 404.025445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 408.300617] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 417.281027] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 422.170531] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 427.893177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 428.068357] Lustre: lustre-OST0000: new disk, initializing [ 428.071621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 428.076709] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 428.161547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 429.356560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 429.365226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 429.434238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 432.649877] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 440.332284] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 443.868430] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.080474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.234940] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 448.418872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 450.023145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 450.027261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.051702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 450.058508] Lustre: Skipped 1 previous similar message [ 460.259889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 460.266060] Lustre: Skipped 3 previous similar messages [ 464.351337] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 464.529173] Lustre: server umount lustre-OST0000 complete [ 470.503080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.511201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 470.514630] Lustre: Skipped 1 previous similar message [ 470.527811] LustreError: Skipped 1 previous similar message [ 480.740466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 480.747017] Lustre: Skipped 5 previous similar messages [ 482.271197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 482.439626] Lustre: server umount lustre-MDT0000 complete [ 485.860590] LustreError: 15086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.901552] LustreError: 15086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 485.961661] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574466 with bad export cookie 16075726531865701476 [ 485.963489] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.980243] LustreError: 15066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 486.341297] Lustre: server umount lustre-MDT0001 complete [ 490.818246] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:27:49 (1787574469) [ 493.516923] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 495.778803] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:27:54 (1787574474) [ 504.050315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.791527] LustreError: 19457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.874296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.921181] LustreError: 19458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.033285] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 515.053595] LustreError: 19457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.740138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.889254] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 531.689678] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 537.309560] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 543.713183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 551.095742] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 559.263801] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 570.855542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 570.864216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.877035] Lustre: Skipped 1 previous similar message [ 570.882504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 576.347332] Lustre: server umount lustre-OST0000 complete [ 580.578562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 580.589141] LustreError: Skipped 1 previous similar message [ 580.594090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.615693] Lustre: Skipped 1 previous similar message [ 586.758992] Lustre: server umount lustre-MDT0000 complete [ 589.285494] LustreError: 19452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.315262] LustreError: 19452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 590.880659] LustreError: 19438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574571 with bad export cookie 16075726531865702358 [ 590.887382] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 590.901104] LustreError: 19438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 591.473718] Lustre: server umount lustre-MDT0001 complete [ 605.299297] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 608.183737] Key type lgssc unregistered [ 608.493858] LNet: 22993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.499111] LNetError: 22993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.509370] LNet: Removed LNI 192.168.206.140@tcp [ 609.633244] Key type .llcrypt unregistered [ 609.635490] Key type ._llcrypt unregistered [ 629.714503] Key type ._llcrypt registered [ 629.717199] Key type .llcrypt registered [ 629.787946] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 630.403257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 630.439925] alg: No test for adler32 (adler32-zlib) [ 631.497822] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 631.727823] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 633.424043] Key type lgssc registered [ 634.634060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 644.091819] loop: module loaded [ 644.231303] loop0: detected capacity change from 0 to 204800000 [ 644.512032] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 644.530847] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.166024] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.900812] loop0: detected capacity change from 0 to 204800000 [ 652.132544] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 652.141692] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 652.154360] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 657.860242] loop0: detected capacity change from 0 to 204800000 [ 662.400981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 662.416534] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.657944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 663.742637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 663.862752] Lustre: lustre-MDT0000: new disk, initializing [ 663.985179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 664.034976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 668.077387] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 676.099301] loop1: detected capacity change from 0 to 204800000 [ 680.505966] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 680.765491] Lustre: lustre-OST07c6: new disk, initializing [ 680.774936] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 680.784034] Lustre: Skipped 1 previous similar message [ 680.788220] Lustre: 26176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 680.848907] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 681.788966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 681.798913] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 681.832559] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 686.196452] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 691.397390] Lustre: 26183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 691.407151] Lustre: 26183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 691.413219] Lustre: 26183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 691.418196] Lustre: 26183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 691.425817] Lustre: 26183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 691.435042] Lustre: 26183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 711.897276] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 711.905453] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 711.914399] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 711.921301] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 711.927524] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 711.935915] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.535697] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 716.549540] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 716.556995] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 716.574041] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.590895] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 716.596943] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.600619] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 716.605311] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.610902] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 716.616892] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.621152] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.625426] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.928288] Lustre: 26183:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 719.934265] Lustre: 26183:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 719.939111] Lustre: 26183:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 719.955144] Lustre: 26183:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.966053] Lustre: 26183:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 719.983442] Lustre: 26183:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.008738] Lustre: 26183:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 720.024807] Lustre: 26183:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.032695] Lustre: 26183:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 720.040145] Lustre: 26183:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.054985] Lustre: 26183:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.069345] Lustre: 26183:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.020926] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 725.026756] Lustre: 25162:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 725.034143] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 725.040152] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.044515] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.048991] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.053113] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 725.057674] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.062068] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.066520] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.070694] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.075194] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.739751] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 734.757964] Lustre: 25164:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 734.768584] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 734.777838] Lustre: 25164:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 734.785441] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 734.794766] Lustre: 25164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 734.801383] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 734.809752] Lustre: 25164:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 734.816395] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 734.823661] Lustre: 25164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 734.836160] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 734.843399] Lustre: 25164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 789.475792] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 789.479581] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.487931] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 794.592736] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 799.200246] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 799.367636] Lustre: server umount lustre-OST07c6 complete [ 808.605650] Lustre: server umount lustre-MDT0000 complete [ 823.151273] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 825.473241] Key type lgssc unregistered [ 825.725732] LNet: 28738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.738257] LNetError: 28738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.790870] LNet: Removed LNI 192.168.206.140@tcp [ 827.726900] Key type .llcrypt unregistered [ 827.730503] Key type ._llcrypt unregistered [ 848.410930] Key type ._llcrypt registered [ 848.412345] Key type .llcrypt registered [ 848.515512] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 849.552345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 849.771499] alg: No test for adler32 (adler32-zlib) [ 850.867944] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 851.113468] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 852.767276] Key type lgssc registered [ 854.049592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 861.688477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 861.706593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.146833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.129819] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 872.430537] LustreError: 30010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.459500] LustreError: 30010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 876.416754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.774945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 881.405876] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 889.177332] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 894.800637] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 900.433849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 900.805945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.506841] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 908.017198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 913.677221] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 927.714082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 927.720052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.734093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 928.737400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 928.741779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 928.755371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.782041] Lustre: Skipped 1 previous similar message [ 933.453280] Lustre: server umount lustre-OST0000 complete [ 938.464900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 938.472777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 938.491688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 944.042880] Lustre: server umount lustre-MDT0000 complete [ 944.098730] LustreError: 30010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 944.121800] LustreError: 30010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 947.550815] LustreError: 29989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574928 with bad export cookie 6608810656655813952 [ 947.555177] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 947.559918] LustreError: 29989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 947.895504] Lustre: server umount lustre-MDT0001 complete [ 958.895590] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 961.428205] Key type lgssc unregistered [ 961.680491] LNet: 33748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.684959] LNetError: 33748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.693523] LNet: Removed LNI 192.168.206.140@tcp [ 962.422301] Key type .llcrypt unregistered [ 962.426593] Key type ._llcrypt unregistered [ 978.532566] Key type ._llcrypt registered [ 978.534461] Key type .llcrypt registered [ 978.617275] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 979.558863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 979.591533] alg: No test for adler32 (adler32-zlib) [ 980.686432] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 980.914965] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 982.633760] Key type lgssc registered [ 983.720979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 990.716876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 990.725782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.084971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 995.846543] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1001.449128] LustreError: 34996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.465186] LustreError: 34996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1003.611599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.971479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1008.588408] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1015.966850] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1022.293143] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1029.479312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1029.808636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.803922] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1037.021331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1044.089483] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1058.790589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.793722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1058.793792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1058.803146] Lustre: Skipped 1 previous similar message [ 1058.815254] Lustre: Skipped 1 previous similar message [ 1063.908690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1063.921552] Lustre: Skipped 1 previous similar message [ 1064.449537] Lustre: server umount lustre-OST0000 complete [ 1069.029515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.045393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1069.053604] Lustre: Skipped 1 previous similar message [ 1074.146562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1074.156421] Lustre: Skipped 1 previous similar message [ 1079.264333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.273523] Lustre: Skipped 1 previous similar message [ 1082.335117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1082.509924] Lustre: server umount lustre-MDT0000 complete [ 1084.385892] LustreError: 34995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.401631] LustreError: 34995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1086.049516] LustreError: 35731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575066 with bad export cookie 13919878252979089743 [ 1086.067055] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1086.075439] LustreError: 35731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1086.445332] Lustre: server umount lustre-MDT0001 complete [ 1096.743974] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1099.360697] Key type lgssc unregistered [ 1099.678992] LNet: 38554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.692606] LNetError: 38554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.711279] LNet: Removed LNI 192.168.206.140@tcp [ 1100.767780] Key type .llcrypt unregistered [ 1100.769800] Key type ._llcrypt unregistered [ 1124.360673] Key type ._llcrypt registered [ 1124.362504] Key type .llcrypt registered [ 1124.452405] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1125.097397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1125.183712] alg: No test for adler32 (adler32-zlib) [ 1126.171752] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 1126.425718] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1128.071155] Key type lgssc registered [ 1128.937830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1136.140529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1136.163838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.575483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.191958] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1147.877492] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.887870] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1150.187221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.803167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1156.420060] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1164.610878] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1170.539586] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1176.979220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1177.263551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1182.393831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1182.793696] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1189.659382] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1201.136122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.139286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1201.165566] Lustre: Skipped 2 previous similar messages [ 1201.180407] Lustre: Skipped 2 previous similar messages [ 1202.580355] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 1205.606260] Lustre: server umount lustre-MDT0000 complete [ 1206.246492] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.266117] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1207.685032] LustreError: 39967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.282139] LustreError: 40705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575190 with bad export cookie 166740936070768295 [ 1209.298761] LustreError: 40705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1209.304524] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1211.360562] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.362269] LustreError: 42464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1211.387490] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1214.369243] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1214.374494] Lustre: Skipped 1 previous similar message [ 1215.828637] Lustre: server umount lustre-MDT0001 complete [ 1236.507728] Lustre: server umount lustre-OST0000 complete [ 1252.784863] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1255.029368] Key type lgssc unregistered [ 1255.291462] LNet: 43835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.297176] LNetError: 43835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.320545] LNet: Removed LNI 192.168.206.140@tcp [ 1256.080075] Key type .llcrypt unregistered [ 1256.085566] Key type ._llcrypt unregistered [ 1274.882313] Key type ._llcrypt registered [ 1274.885181] Key type .llcrypt registered [ 1274.991537] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1275.709409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1275.802711] alg: No test for adler32 (adler32-zlib) [ 1276.865514] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 1277.066211] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1278.751141] Key type lgssc registered [ 1279.740047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1286.481610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1286.496231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.886850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.885679] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1297.381263] LustreError: 45231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1297.414151] LustreError: 45231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1299.346502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1299.852287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1303.313188] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1309.627655] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1315.393974] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1323.338573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1323.866869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.639502] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1333.116681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1338.323373] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1345.506138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1345.515259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.539980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1346.456737] Lustre: lustre-OST0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 1348.579942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1348.581501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1348.583200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1351.534834] Lustre: server umount lustre-OST0000 complete [ 1371.617105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1371.631893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1371.643670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1371.647127] Lustre: Skipped 1 previous similar message [ 1373.730940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.784888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1378.796039] Lustre: Skipped 3 previous similar messages [ 1383.391220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1383.749357] Lustre: server umount lustre-MDT0000 complete [ 1383.912475] LustreError: 46325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.928332] LustreError: 46325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1387.289091] LustreError: 45211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575368 with bad export cookie 2003250420687986994 [ 1387.291308] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1387.303556] LustreError: 45211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1393.724543] Lustre: server umount lustre-MDT0001 complete [ 1404.616496] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1407.607719] Key type lgssc unregistered [ 1407.967265] LNet: 48945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.978715] LNetError: 48945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1408.002946] LNet: Removed LNI 192.168.206.140@tcp [ 1408.854843] Key type .llcrypt unregistered [ 1408.856879] Key type ._llcrypt unregistered [ 1430.315915] Key type ._llcrypt registered [ 1430.318489] Key type .llcrypt registered [ 1430.456567] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1431.575044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1431.599982] alg: No test for adler32 (adler32-zlib) [ 1432.676396] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 1432.930750] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1434.583215] Key type lgssc registered [ 1435.592603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1442.146147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1442.159933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.521267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.590875] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1454.048476] LustreError: 50351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.090775] LustreError: 50351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1456.022373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.318679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1461.180565] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1468.531985] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1474.511339] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1481.290973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1481.636816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1487.168620] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1488.831802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1494.010446] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1508.986777] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1512.243964] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1524.523934] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-24 8h45m02s [ 1525.801215] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1527.690915] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 1527.782467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.792329] Lustre: Skipped 2 previous similar messages [ 1528.702939] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 1528.708720] Lustre: Skipped 3 previous similar messages [ 1533.523612] Lustre: server umount lustre-MDT0000 complete [ 1533.759218] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.952691] LustreError: 50352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.967143] LustreError: 50352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1536.220338] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1539.972224] LustreError: 50348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1542.681956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.874701] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1543.288487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.379923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1544.316367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1547.612212] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1550.802612] Lustre: DEBUG MARKER: Wait for df (36732) ... [ 1552.684807] Lustre: DEBUG MARKER: done [ 1558.496985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1558.499699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1558.510478] LustreError: Skipped 1 previous similar message [ 1558.512099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1558.520400] Lustre: Skipped 1 previous similar message [ 1558.528873] Lustre: Skipped 4 previous similar messages [ 1563.468703] Lustre: server umount lustre-OST0000 complete [ 1570.272552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1570.277347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1570.282207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1570.283211] Lustre: Skipped 1 previous similar message [ 1570.288249] Lustre: Skipped 3 previous similar messages [ 1572.959710] Lustre: server umount lustre-MDT0000 complete [ 1576.500286] LustreError: 52196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575557 with bad export cookie 3251257039089138301 [ 1576.506304] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1576.518213] LustreError: 52196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1576.857066] Lustre: server umount lustre-MDT0001 complete [ 1588.624249] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1591.122180] Key type lgssc unregistered [ 1591.313733] LNet: 55303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.322753] LNetError: 55303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.336094] LNet: Removed LNI 192.168.206.140@tcp [ 1592.181183] Key type .llcrypt unregistered [ 1592.185250] Key type ._llcrypt unregistered [ 1632.225933] Key type ._llcrypt registered [ 1632.227721] Key type .llcrypt registered [ 1632.333328] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1633.474922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1633.507181] alg: No test for adler32 (adler32-zlib) [ 1634.532614] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 1634.720340] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1636.375203] Key type lgssc registered [ 1637.462907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1644.556442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1644.581315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.820771] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1645.836370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1645.936257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.966555] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1657.319868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.393977] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1657.411820] Lustre: 57521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1657.623442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1661.729130] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1670.662846] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1678.051476] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1684.727720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1684.983531] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1685.199625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1690.155288] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1693.826630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1697.105404] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1704.612552] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-24 8h48m03s [ 1706.530385] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1714.986919] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1717.329375] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1717.462872] LustreError: 60451:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1737.642781] Lustre: lustre-MDT0000: Client ec932ec3-4ca3-4f43-957b-064d13901f24 (at 192.168.206.40@tcp) reconnecting [ 1757.543117] LustreError: 60451:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1759.622076] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1762.024210] Lustre: DEBUG MARKER: done [ 1770.238733] Lustre: server umount lustre-OST0000 complete [ 1775.588651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1775.607949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1775.624105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1776.688162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1776.701093] Lustre: Skipped 2 previous similar messages [ 1782.768871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.847234] Lustre: 56843:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787575751/real 1787575751] req@00000000708eaf41 x1874408888128768/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787575767 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1786.883220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.899663] Lustre: Skipped 1 previous similar message [ 1787.937158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1787.945277] Lustre: Skipped 2 previous similar messages [ 1788.384369] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1788.590289] Lustre: server umount lustre-MDT0000 complete [ 1792.033051] Lustre: 56844:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787575756/real 1787575756] req@000000004e30b7be x1874408888131072/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787575772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1792.458130] LustreError: 58271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575773 with bad export cookie 8922286367248941619 [ 1792.461233] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1792.473454] LustreError: 58271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1792.797635] Lustre: server umount lustre-MDT0001 complete [ 1802.954674] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1806.193094] Key type lgssc unregistered [ 1806.499815] LNet: 61854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.516168] LNetError: 61854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.535515] LNet: Removed LNI 192.168.206.140@tcp [ 1807.479457] Key type .llcrypt unregistered [ 1807.482370] Key type ._llcrypt unregistered [ 1850.203948] Key type ._llcrypt registered [ 1850.205821] Key type .llcrypt registered [ 1850.317947] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1851.127724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1851.179196] alg: No test for adler32 (adler32-zlib) [ 1852.206487] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 1852.376586] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1854.047154] Key type lgssc registered [ 1854.775655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1858.300798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1858.326847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.619214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1859.654069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1859.830807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.800650] Lustre: Failing over lustre-MDT0000 [ 1861.229040] Lustre: server umount lustre-MDT0000 complete [ 1869.067857] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:50:48 (1787575848) [ 1877.328577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.743741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.110289] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1888.233911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.309484] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1888.340780] Lustre: 64652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1888.547363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1892.697757] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1901.316915] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1907.733744] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1915.088654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1915.268544] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1915.457042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1921.751463] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1924.771366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1930.205222] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1936.081228] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1945.063174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1945.073660] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.092579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1950.554651] Lustre: server umount lustre-MDT0000 complete [ 1954.528812] LustreError: 64645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575935 with bad export cookie 353704993462453498 [ 1954.537450] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1954.539777] LustreError: 64645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1954.788684] Lustre: server umount lustre-MDT0001 complete [ 1958.614333] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1963.453851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1971.169132] Lustre: 63578:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787575936/real 1787575936] req@00000000c1659722 x1874409116704768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787575952 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1971.223488] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.231953] Lustre: Skipped 2 previous similar messages [ 1972.437432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.263310] Lustre: 63580:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787575940/real 1787575940] req@0000000011bcb615 x1874409116704896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787575956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1976.288886] Lustre: 63579:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787575941/real 1787575941] req@00000000832a39ea x1874409116705024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787575957 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1979.423830] LustreError: 63576:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000069bff3cc x1874409116705792/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1979.708421] LustreError: 66502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.726137] LustreError: 66502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1979.804155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.917134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1980.767257] Lustre: 63578:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787575945/real 1787575945] req@0000000080faa093 x1874409116705280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787575961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1980.832694] LustreError: 66503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.857331] LustreError: 68420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.887331] LustreError: 68420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1984.035431] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1987.041356] Lustre: 63578:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787575951/real 1787575951] req@0000000035f09f3e x1874409116705408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787575967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1987.046550] LustreError: 68419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1987.077525] LustreError: 68419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1992.174106] LustreError: 68421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.191954] LustreError: 68421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1992.486617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.784947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1997.262357] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1997.813228] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2000.939557] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2005.084994] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2010.932156] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2018.276605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2018.285446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.317285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2018.319763] Lustre: Skipped 2 previous similar messages [ 2021.344164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2021.346707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.359220] Lustre: Skipped 1 previous similar message [ 2021.368312] Lustre: Skipped 1 previous similar message [ 2023.931819] Lustre: server umount lustre-MDT0000 complete [ 2026.473507] LustreError: 68421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.503441] LustreError: 68421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2027.716813] LustreError: 64646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576008 with bad export cookie 353704993462455773 [ 2027.727965] LustreError: 64646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2027.731541] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.273165] Lustre: server umount lustre-MDT0001 complete [ 2035.115838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2044.415942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.968167] Lustre: 63579:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787576012/real 1787576012] req@00000000c8afeb75 x1874409116732672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787576028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2048.001285] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.443904] LustreError: 66502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.553511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.611428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2057.318751] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2065.845881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.716209] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2071.527172] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2077.672390] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2078.309343] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.602529] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2092.004023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2092.013666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.026780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2092.035473] Lustre: Skipped 1 previous similar message [ 2096.806853] Lustre: server umount lustre-MDT0000 complete [ 2098.147448] LustreError: 71103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.169450] LustreError: 71103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2100.146638] LustreError: 64645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576081 with bad export cookie 353704993462457152 [ 2100.149073] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2100.173354] LustreError: 64645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2100.478439] Lustre: server umount lustre-MDT0001 complete [ 2107.071428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2110.547291] Lustre: DEBUG MARKER: files compared the same [ 2123.284545] Lustre: server umount lustre-OST0000 complete [ 2140.084257] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2142.876512] Key type lgssc unregistered [ 2143.217313] LNet: 74382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.227726] LNetError: 74382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2143.252883] LNet: Removed LNI 192.168.206.140@tcp [ 2144.119489] Key type .llcrypt unregistered [ 2144.122455] Key type ._llcrypt unregistered [ 2169.911790] Key type ._llcrypt registered [ 2169.914894] Key type .llcrypt registered [ 2170.091861] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2171.590590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2171.645764] alg: No test for adler32 (adler32-zlib) [ 2172.755749] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 2172.960606] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2174.623198] Key type lgssc registered [ 2175.757578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2184.640188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2184.668455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.328282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.165817] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug +malloc all [ 2196.448452] LustreError: 75803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2196.478208] LustreError: 75803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2199.237923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.642976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.251671] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug +malloc all [ 2213.568484] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2221.010793] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2229.020661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2229.270802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.338207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2235.090781] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug +malloc all [ 2243.962376] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2254.817497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.817513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2254.818640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2254.844299] Lustre: Skipped 1 previous similar message [ 2259.671840] Lustre: server umount lustre-OST0000 complete [ 2265.057831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.059799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2265.076976] Lustre: Skipped 1 previous similar message [ 2265.087872] Lustre: Skipped 2 previous similar messages [ 2269.532271] Lustre: server umount lustre-MDT0000 complete [ 2270.176231] LustreError: 75804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.185462] LustreError: 75804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2273.199053] LustreError: 75784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576254 with bad export cookie 2872819412847729115 [ 2273.211091] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2273.697741] Lustre: server umount lustre-MDT0001 complete [ 2285.525445] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2287.884358] Key type lgssc unregistered [ 2288.160562] LNet: 79337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.181167] LNetError: 79337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.204430] LNet: Removed LNI 192.168.206.140@tcp [ 2289.049564] Key type .llcrypt unregistered [ 2289.051210] Key type ._llcrypt unregistered [ 2317.356465] Key type ._llcrypt registered [ 2317.357970] Key type .llcrypt registered [ 2317.419328] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2318.167544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.233977] alg: No test for adler32 (adler32-zlib) [ 2319.441211] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 2319.724206] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2321.431200] Key type lgssc registered [ 2322.934644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2330.815318] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2330.851097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2346.463483] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2351.583325] LustreError: 80728:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 2378.271209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.481609] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2392.109442] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2400.432288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.639737] Lustre: *** cfs_fail_loc=706, val=0*** [ 2401.059771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.170823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2405.585762] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2406.368890] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.391618] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2411.494887] LustreError: 81842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.519755] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.542787] LustreError: 80754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2415.487690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2415.941499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2420.620839] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2429.109508] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2434.790298] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2437.091440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2437.104498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.129641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.706176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2441.706192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.708610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2442.987738] Lustre: server umount lustre-OST0000 complete [ 2451.949231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.953409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2451.973072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2452.002211] Lustre: Skipped 2 previous similar messages [ 2453.333534] Lustre: server umount lustre-MDT0000 complete [ 2457.057979] LustreError: 81820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.081576] LustreError: 81820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2457.109479] LustreError: 80734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576437 with bad export cookie 7525543892823790493 [ 2457.120878] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2457.124076] LustreError: 80734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2457.434903] Lustre: server umount lustre-MDT0001 complete [ 2469.755319] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2472.637828] Key type lgssc unregistered [ 2472.907070] LNet: 84348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.921066] LNetError: 84348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2472.943276] LNet: Removed LNI 192.168.206.140@tcp [ 2473.846806] Key type .llcrypt unregistered [ 2473.849502] Key type ._llcrypt unregistered [ 2498.016958] Key type ._llcrypt registered [ 2498.019836] Key type .llcrypt registered [ 2498.155477] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2499.308421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2499.320103] alg: No test for adler32 (adler32-zlib) [ 2500.377297] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 2500.653224] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2502.343288] Key type lgssc registered [ 2503.584952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2511.859603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2511.876142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.485208] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2528.362938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.190313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2534.371528] LustreError: 86466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.410903] LustreError: 86466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2536.377613] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2538.469729] LustreError: 86470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.498613] LustreError: 86470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2543.586921] LustreError: 86469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.613118] LustreError: 86469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2547.207371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2547.558924] LustreError: 86464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.568206] LustreError: 86464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2547.581546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2552.514366] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2552.806041] LustreError: 86470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.816759] LustreError: 86470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2561.556582] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2561.570251] Lustre: MGS: Client 32f69ad0-2e92-4736-83cf-3966e0508fc5 (at 0@lo) reconnecting [ 2561.615661] LustreError: 86480:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2561.897686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.996113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2566.246625] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2582.506232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2582.512386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2582.520021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2582.552103] Lustre: Skipped 1 previous similar message [ 2585.650872] Lustre: server umount lustre-OST0000 complete [ 2592.738063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.738817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2592.752412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.773425] Lustre: Skipped 2 previous similar messages [ 2595.510349] Lustre: server umount lustre-MDT0000 complete [ 2597.857292] LustreError: 86469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.881463] LustreError: 86469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2600.334309] Lustre: server umount lustre-MDT0001 complete [ 2604.750618] Lustre: server umount MGS complete [ 2617.498058] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:03:16 (1787576596) [ 2653.642097] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 2663.012180] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2702.456314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.389695] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2716.824233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2717.077820] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2717.137956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2717.198515] Lustre: lustre-MDT0001: new disk, initializing [ 2717.303444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2717.358697] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2717.376453] Lustre: 93636:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2717.388405] LustreError: 93636:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2722.502373] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2732.476557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2732.774279] Lustre: lustre-OST0000: new disk, initializing [ 2732.778542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2732.788668] Lustre: 94403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2732.856659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2740.504439] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2742.376870] Lustre: 94785:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000036b809c4 x1874409796182784/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787576739 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2751.270854] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2758.864328] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2758.882774] Lustre: MGS: Client 85806fa1-0c50-4aea-9380-6fb4852c04de (at 0@lo) reconnecting [ 2758.950964] Lustre: 92906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2758.975213] LustreError: 93622:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2758.999119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2759.067650] Lustre: lustre-MDT0000: new disk, initializing [ 2759.160518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2759.182587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2759.193422] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2760.235271] Lustre: 95472:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000091b6a61d x1874409796189824/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787576757 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2764.575980] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2765.413493] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2769.320292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2769.328250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2769.466145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2798.053406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.054795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2798.063783] Lustre: Skipped 2 previous similar messages [ 2803.175108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2803.187566] Lustre: Skipped 2 previous similar messages [ 2803.439919] Lustre: server umount lustre-OST0000 complete [ 2808.290474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2808.295093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2808.311483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2808.319700] Lustre: Skipped 1 previous similar message [ 2808.326471] Lustre: Skipped 1 previous similar message [ 2813.574921] Lustre: server umount lustre-MDT0000 complete [ 2832.351260] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2832.491368] Lustre: server umount lustre-MDT0001 complete [ 2835.686394] Lustre: server umount MGS complete [ 2846.547815] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:07:05 (1787576825) [ 2864.997710] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2867.767221] Key type lgssc unregistered [ 2868.096840] LNet: 97975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2868.116669] LNetError: 97975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2869.159398] LNet: Removed LNI 192.168.206.140@tcp [ 2869.918374] Key type .llcrypt unregistered [ 2869.920357] Key type ._llcrypt unregistered [ 2887.317603] Key type ._llcrypt registered [ 2887.320435] Key type .llcrypt registered [ 2887.425770] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2888.449345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2888.479662] alg: No test for adler32 (adler32-zlib) [ 2889.516579] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 2889.788272] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2891.464222] Key type lgssc registered [ 2892.544901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2900.231725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2900.249903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.883192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.600487] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2911.207338] LustreError: 99198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.238064] LustreError: 99198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2915.562052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.869493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2920.689855] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2928.474774] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2933.726346] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2941.176981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.684410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.804595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2947.813097] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2956.180452] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2971.622622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.626895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2971.627284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.634740] Lustre: Skipped 1 previous similar message [ 2975.021925] Lustre: server umount lustre-OST0000 complete [ 2981.858193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.862782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2981.888131] Lustre: Skipped 1 previous similar message [ 2981.900702] Lustre: Skipped 2 previous similar messages [ 2984.898950] Lustre: server umount lustre-MDT0000 complete [ 2986.976471] LustreError: 99198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.001175] LustreError: 99198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2988.406976] LustreError: 99178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576969 with bad export cookie 12495239907678661068 [ 2988.410597] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2988.420856] LustreError: 99178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2988.735726] Lustre: server umount lustre-MDT0001 complete [ 3000.548870] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 3003.648885] Key type lgssc unregistered [ 3003.911755] LNet: 102754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3003.917416] LNetError: 102754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3003.928861] LNet: Removed LNI 192.168.206.140@tcp [ 3004.929515] Key type .llcrypt unregistered [ 3004.933691] Key type ._llcrypt unregistered [ 3026.116854] Key type ._llcrypt registered [ 3026.119294] Key type .llcrypt registered [ 3026.263595] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3027.682364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3027.822161] alg: No test for adler32 (adler32-zlib) [ 3028.929408] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 3029.496184] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 3031.295204] Key type lgssc registered [ 3032.678314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3038.641063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3038.648578] LustreError: 103839:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3038.684545] LustreError: 103843:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3038.693722] LustreError: 103839:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4961 [ 3038.700931] LustreError: 103839:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3038.713425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.715498] LustreError: 103839:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3040.275334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.172884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3047.483968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3052.073634] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3056.847416] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing lsmod [ 3059.609077] LustreError: 105048:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3059.635951] LustreError: 105051:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3059.643455] LustreError: 105048:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4978 [ 3059.647840] LustreError: 105048:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3059.657039] LustreError: 105048:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3059.663612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3059.976113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.753107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3062.763185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.779216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.323330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.324435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.353403] Lustre: Skipped 1 previous similar message [ 3069.009142] Lustre: server umount lustre-MDT0000 complete [ 3070.444830] LustreError: 103881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.466027] LustreError: 103881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3072.487582] LustreError: 104473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3072.507023] LustreError: 104473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3073.196574] LustreError: 103861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577054 with bad export cookie 5146142177724463106 [ 3073.215153] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3074.655369] LustreError: 104312:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003f72cdb3 x1874410350858112/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3074.772983] Lustre: server umount lustre-MDT0001 complete [ 3088.415947] Lustre: server umount lustre-OST0000 complete [ 3096.709253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.085744] LustreError: 106114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3097.136877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.364480] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3102.177937] LustreError: 106115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.274766] LustreError: 106114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.595785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.921155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3114.419386] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3121.948275] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3126.710971] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3133.853388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3134.506434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3141.428443] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3141.634028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3151.266530] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3160.033959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3160.052053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3160.110307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3160.123052] Lustre: Skipped 2 previous similar messages [ 3162.082064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3162.097744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.110817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3162.114441] Lustre: Skipped 1 previous similar message [ 3164.773531] Lustre: server umount lustre-OST0000 complete [ 3171.296483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3171.310401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.333511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3174.542179] Lustre: server umount lustre-MDT0000 complete [ 3177.440343] LustreError: 108611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.455651] LustreError: 108611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3179.105566] LustreError: 106094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577159 with bad export cookie 5146142177724463806 [ 3179.122443] LustreError: 106094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3179.132213] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3179.307264] Lustre: server umount lustre-MDT0001 complete [ 3190.855395] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 3193.616447] Key type lgssc unregistered [ 3193.856350] LNet: 109648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3193.863293] LNetError: 109648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3193.875533] LNet: Removed LNI 192.168.206.140@tcp [ 3194.631532] Key type .llcrypt unregistered [ 3194.634617] Key type ._llcrypt unregistered [ 3219.407490] Key type ._llcrypt registered [ 3219.412363] Key type .llcrypt registered [ 3219.594493] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3221.102967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3221.156713] alg: No test for adler32 (adler32-zlib) [ 3222.337021] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 3222.561456] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 3224.271146] Key type lgssc registered [ 3225.330349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3233.170395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3233.188533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3234.678685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.404679] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3245.029965] LustreError: 111067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3245.069693] LustreError: 111067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3248.538981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3249.003361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3252.669932] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3259.699625] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3264.732712] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3272.101622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.429531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.381125] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3279.598124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3286.478421] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3292.311372] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3295.956378] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3300.320645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3300.328982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.337080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3300.351550] Lustre: Skipped 1 previous similar message [ 3300.360251] Lustre: Skipped 1 previous similar message [ 3305.451187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.463171] Lustre: Skipped 1 previous similar message [ 3305.859038] Lustre: server umount lustre-OST0000 complete [ 3310.561567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3310.570867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3310.590503] Lustre: Skipped 1 previous similar message [ 3310.617505] Lustre: Skipped 1 previous similar message [ 3315.681136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.689443] Lustre: Skipped 1 previous similar message [ 3317.124991] Lustre: server umount lustre-MDT0000 complete [ 3320.800602] LustreError: 112134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.822366] LustreError: 112134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3321.401379] LustreError: 111047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577302 with bad export cookie 13963018986844393347 [ 3321.409309] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3321.413557] LustreError: 111047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3321.820953] Lustre: server umount lustre-MDT0001 complete [ 3333.225794] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 3335.536990] Key type lgssc unregistered [ 3335.761966] LNet: 114708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3335.769500] LNetError: 114708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3335.795584] LNet: Removed LNI 192.168.206.140@tcp [ 3336.648230] Key type .llcrypt unregistered [ 3336.650675] Key type ._llcrypt unregistered [ 3354.431846] Key type ._llcrypt registered [ 3354.435664] Key type .llcrypt registered [ 3354.549935] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3355.642494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3355.697082] alg: No test for adler32 (adler32-zlib) [ 3357.015331] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 3357.264302] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 3358.919413] Key type lgssc registered [ 3360.180910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3368.598482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3368.613990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.160602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.079591] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3380.716635] LustreError: 115932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3380.745128] LustreError: 115932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3384.072124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3384.459017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3389.379524] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3397.862264] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3403.811870] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3410.756330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3411.061559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3415.162990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3417.157253] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3425.345279] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3441.125614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3441.143519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.177231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.218288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3445.226681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.233835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.289454] Lustre: Skipped 1 previous similar message [ 3446.779299] Lustre: server umount lustre-OST0000 complete [ 3451.362696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3451.378938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.403663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3455.457067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3455.458502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.625139] Lustre: server umount lustre-MDT0000 complete [ 3460.582365] LustreError: 117490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.606959] LustreError: 117490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3462.404186] LustreError: 115912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577443 with bad export cookie 9534219346632112332 [ 3462.410596] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3462.415260] LustreError: 115912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3462.885663] Lustre: server umount lustre-MDT0001 complete [ 3474.694868] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 3477.844691] Key type lgssc unregistered [ 3478.103060] LNet: 119491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3478.116712] LNetError: 119491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3478.134340] LNet: Removed LNI 192.168.206.140@tcp [ 3478.939271] Key type .llcrypt unregistered [ 3478.941971] Key type ._llcrypt unregistered [ 3504.236800] Key type ._llcrypt registered [ 3504.239221] Key type .llcrypt registered [ 3504.402750] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3505.528135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3505.581178] alg: No test for adler32 (adler32-zlib) [ 3506.600413] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 3506.793615] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 3508.448448] Key type lgssc registered [ 3509.838683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3518.919950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3518.954625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.529663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.262596] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3530.727168] LustreError: 120913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3530.745719] LustreError: 120913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3535.851974] LustreError: 120914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.876919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.386196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3542.597306] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3552.790790] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3561.656996] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3569.823406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3570.627860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3576.696162] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3585.526267] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3591.184707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3598.040973] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3601.752466] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3601.758281] Lustre: Skipped 2 previous similar messages [ 3606.427364] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3606.432827] Lustre: Skipped 2 previous similar messages [ 3607.190692] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3607.241750] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3617.213988] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.40@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3617.221411] Lustre: Skipped 2 previous similar messages [ 3624.526271] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.40@tcp 192.168.0.[2,10]@tcp [ 3628.610498] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.40/24@tcp in log lustre-MDT0000 [ 3628.622449] Lustre: Skipped 2 previous similar messages [ 3630.507947] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.40/24@tcp [ 3630.521032] Lustre: Skipped 1 previous similar message [ 3641.828441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3641.835799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.855357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3642.345312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3642.358378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3642.361647] Lustre: Skipped 1 previous similar message [ 3642.767584] Lustre: server umount lustre-OST0000 complete [ 3647.461783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.464863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3647.496664] Lustre: Skipped 1 previous similar message [ 3652.553488] Lustre: server umount lustre-MDT0000 complete [ 3652.589228] LustreError: 122188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3652.610360] LustreError: 122188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3656.453778] LustreError: 120894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577637 with bad export cookie 8218362960832681275 [ 3656.461989] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3656.470145] LustreError: 120894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3656.917291] Lustre: server umount lustre-MDT0001 complete [ 3666.183424] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 3668.471309] Key type lgssc unregistered [ 3668.872714] LNet: 125660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3668.879842] LNetError: 125660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3668.890317] LNet: Removed LNI 192.168.206.140@tcp [ 3669.672887] Key type .llcrypt unregistered [ 3669.678889] Key type ._llcrypt unregistered [ 3702.356641] Key type ._llcrypt registered [ 3702.360300] Key type .llcrypt registered [ 3702.469479] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3704.414159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3704.445069] alg: No test for adler32 (adler32-zlib) [ 3705.678570] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 3705.988730] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 3707.720061] Key type lgssc registered [ 3708.997368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3716.540508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3716.561435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.019931] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3718.042232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3718.051674] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.40/24@tcp [ 3722.038968] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3727.334944] LustreError: 127226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.351690] LustreError: 127226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3731.037422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.216990] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3731.236363] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.40/24@tcp [ 3731.280923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3736.411195] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3745.751326] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3751.185614] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3756.734108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3756.972461] LustreError: 129051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3757.144448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3762.804496] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3763.215385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3770.718171] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3782.643840] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3782.652602] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3782.666602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3783.661104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3783.663365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3783.672939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3788.787517] LustreError: 129053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3788.805445] LustreError: 129053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3788.914318] Lustre: server umount lustre-OST0000 complete [ 3793.889413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3793.919509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.931239] Lustre: Skipped 2 previous similar messages [ 3799.010625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.015408] Lustre: Skipped 1 previous similar message [ 3799.354748] Lustre: server umount lustre-MDT0000 complete [ 3803.819350] LustreError: 127204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577784 with bad export cookie 6264516239145228876 [ 3803.823559] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3803.832703] LustreError: 127204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3804.401604] Lustre: server umount lustre-MDT0001 complete [ 3821.150755] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 3825.948961] Key type lgssc unregistered [ 3826.616658] LNet: 130819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3826.631398] LNetError: 130819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3826.643689] LNet: Removed LNI 192.168.206.140@tcp [ 3827.588538] Key type .llcrypt unregistered [ 3827.596577] Key type ._llcrypt unregistered [ 3856.589908] Key type ._llcrypt registered [ 3856.594246] Key type .llcrypt registered [ 3856.734300] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 3870.972972] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3871.869383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3871.948339] alg: No test for adler32 (adler32-zlib) [ 3872.922660] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 3873.278197] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 3874.999296] Key type lgssc registered [ 3875.982619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.084122] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3914.120883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.523371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3915.611162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3915.801883] Lustre: lustre-MDT0000: new disk, initializing [ 3915.926352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3915.957639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3922.559767] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3934.745573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.888849] Lustre: 134216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3934.977515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3934.981698] Lustre: Skipped 1 previous similar message [ 3935.168201] Lustre: lustre-MDT0001: new disk, initializing [ 3935.340895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3935.376267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3935.380665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3940.919153] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3952.237277] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3958.215840] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3964.040286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3964.290953] Lustre: lustre-OST0000: new disk, initializing [ 3964.295145] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3964.300779] Lustre: 136187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3964.408245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3969.633093] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3973.278794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3973.294185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3973.448680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3979.271330] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3985.474619] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3985.867465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3991.153868] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3991.265393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3994.081579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.084233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3994.090349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.095134] Lustre: Skipped 1 previous similar message [ 3999.213959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3999.227613] Lustre: Skipped 2 previous similar messages [ 4004.331229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.335047] Lustre: Skipped 1 previous similar message [ 4007.391131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4007.639400] Lustre: server umount lustre-OST0000 complete [ 4012.003464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4012.019050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.046827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.564374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.171108] Lustre: server umount lustre-MDT0000 complete [ 4022.406375] LustreError: 136186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578003 with bad export cookie 11005951968206386235 [ 4022.408804] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4022.429450] LustreError: 136186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4022.867667] Lustre: server umount lustre-MDT0001 complete [ 4031.771600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4032.337378] LustreError: 138406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4032.358589] LustreError: 138406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4032.404724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4036.475489] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4037.601916] LustreError: 138407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4042.749246] LustreError: 138406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.665854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4045.973421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4049.774576] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4056.615354] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4062.850866] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4070.561544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4071.000322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4077.355705] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4085.822685] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4092.292650] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4092.450278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4098.665267] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4099.005143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4109.139840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4109.285582] Lustre: lustre-OST0001: new disk, initializing [ 4109.288303] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4109.292345] Lustre: 141778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4109.357810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4116.401794] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4118.113274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4118.126607] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4118.240622] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4125.790280] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4126.051866] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4130.458182] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4130.703630] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4136.599231] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90f3087fa800.ost_server_uuid 50 [ 4138.644658] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90f3087fa800.ost_server_uuid in IDLE state after 0 sec [ 4149.217053] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4149.221894] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.229039] LustreError: Skipped 1 previous similar message [ 4149.240136] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4149.250126] Lustre: Skipped 1 previous similar message [ 4149.272518] Lustre: Skipped 3 previous similar messages [ 4151.453488] Lustre: server umount lustre-OST0001 complete [ 4154.343208] LustreError: 140230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.372488] LustreError: 140230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4157.920198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4157.928778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.943496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4159.462487] LustreError: 140231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.479881] LustreError: 140231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4162.389629] Lustre: server umount lustre-OST0000 complete [ 4168.672959] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4168.683411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4168.698058] Lustre: Skipped 1 previous similar message [ 4173.013131] Lustre: server umount lustre-MDT0000 complete [ 4174.821593] LustreError: 138407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4174.848591] LustreError: 138407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4177.526750] LustreError: 138386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578158 with bad export cookie 11005951968206387117 [ 4177.537046] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4177.539024] LustreError: 138386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4177.936334] Lustre: server umount lustre-MDT0001 complete [ 4199.963193] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4201.690207] Key type lgssc unregistered [ 4201.904962] LNet: 144783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4201.913705] LNetError: 144783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4201.940782] LNet: Removed LNI 192.168.206.140@tcp [ 4202.866684] Key type .llcrypt unregistered [ 4202.869535] Key type ._llcrypt unregistered [ 4258.268958] Key type ._llcrypt registered [ 4258.270948] Key type .llcrypt registered [ 4258.417375] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 4276.610887] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4277.810265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4277.823524] alg: No test for adler32 (adler32-zlib) [ 4278.917589] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 4279.147193] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4280.807520] Key type lgssc registered [ 4282.162290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4331.068285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4331.097977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.514793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4332.542497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4332.619195] Lustre: lustre-MDT0000: new disk, initializing [ 4332.743882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4332.768754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4338.683705] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4352.043354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.158252] Lustre: 148940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4352.228162] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4352.235838] Lustre: Skipped 1 previous similar message [ 4352.304344] Lustre: lustre-MDT0001: new disk, initializing [ 4352.380654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4352.426190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4352.443381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4356.051876] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4365.985042] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4373.000563] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4381.317219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4381.661300] Lustre: lustre-OST0000: new disk, initializing [ 4381.673175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4381.686391] Lustre: 150912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4381.816448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4389.266815] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4391.491905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4391.508054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4391.618636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4399.328350] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4404.705295] Lustre: Failing over lustre-OST0000 [ 4405.039423] Lustre: server umount lustre-OST0000 complete [ 4422.612143] Lustre: 147165:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787578387/real 1787578387] req@0000000012927cc0 x1874411661591040/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787578403 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4422.691444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.650167] Lustre: 147163:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787578388/real 1787578388] req@0000000032ba5b85 x1874411661591936/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787578404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4423.700901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4425.802912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4426.045087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.061878] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4429.087219] Lustre: 147165:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787578393/real 1787578393] req@00000000a40ddf7e x1874411661593216/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787578409 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4429.123889] Lustre: 147165:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4432.171825] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4433.184133] Lustre: 147163:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787578398/real 1787578398] req@000000002dd1e119 x1874411661594368/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787578414 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4433.202960] Lustre: 147163:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4433.209569] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4433.291075] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4433.294333] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4436.924033] Lustre: Failing over lustre-MDT0000 [ 4437.219501] Lustre: server umount lustre-MDT0000 complete [ 4437.281156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.299826] Lustre: Skipped 2 previous similar messages [ 4438.304180] Lustre: 147166:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787578403/real 1787578403] req@000000002990f451 x1874411661595264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787578419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4442.598919] LustreError: 150332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4442.620530] LustreError: 150332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4447.712712] LustreError: 150167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4447.735626] LustreError: 150167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4452.841063] LustreError: 148947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4452.870645] LustreError: 148947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4453.855101] Lustre: 147163:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787578418/real 1787578418] req@0000000039abd239 x1874411661606784/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787578434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4453.881986] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.881211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.237935] LustreError: 152511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4458.271940] LustreError: 152511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4464.099121] LustreError: 147162:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000087b5f81 x1874411661612032/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4464.150800] LustreError: 148947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.185667] LustreError: 148947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4464.438215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.493459] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4468.663683] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4469.641940] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4469.752388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4469.765607] Lustre: Skipped 1 previous similar message [ 4469.801568] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4469.862806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4479.460887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4479.468480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.483388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4482.690404] Lustre: server umount lustre-OST0000 complete [ 4490.208912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4490.220294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4490.231859] Lustre: Skipped 1 previous similar message [ 4490.236170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4490.255361] Lustre: Skipped 2 previous similar messages [ 4492.493443] Lustre: server umount lustre-MDT0000 complete [ 4495.331037] LustreError: 148953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4495.343968] LustreError: 148953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4495.710812] LustreError: 148933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578476 with bad export cookie 13125814033540895508 [ 4495.717078] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4495.726536] LustreError: 148933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4496.110908] Lustre: server umount lustre-MDT0001 complete [ 4508.304555] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4511.719840] Key type lgssc unregistered [ 4512.049573] LNet: 154573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4512.076216] LNetError: 154573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4512.128461] LNet: Removed LNI 192.168.206.140@tcp [ 4513.413319] Key type .llcrypt unregistered [ 4513.417244] Key type ._llcrypt unregistered [ 4556.213406] Key type ._llcrypt registered [ 4556.215823] Key type .llcrypt registered [ 4556.349666] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4557.436075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4557.521667] alg: No test for adler32 (adler32-zlib) [ 4558.489383] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 4558.617553] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4560.231169] Key type lgssc registered [ 4561.150578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4569.743212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4569.769786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.446797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.741508] LustreError: 156096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4576.766818] LustreError: 156096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4577.212921] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4581.861079] LustreError: 156097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.978508] LustreError: 156096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.541829] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4592.100105] LustreError: 156097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4593.756477] LustreError: 156125:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4593.773413] LustreError: 156125:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4593.784360] LustreError: 156125:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4594.552216] Lustre: server umount lustre-MDT0000 complete [ 4611.764117] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 4622.034026] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4669.281380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.580948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4669.636992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4669.718330] Lustre: lustre-MDT0000: new disk, initializing [ 4669.829509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4669.844082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4675.855900] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4687.449075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.543460] Lustre: 159430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4687.589619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4687.596473] Lustre: Skipped 1 previous similar message [ 4687.686982] Lustre: lustre-MDT0001: new disk, initializing [ 4687.780131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4687.814435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4687.824787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4692.849185] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4704.142457] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4710.636222] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4719.051705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4719.270905] Lustre: lustre-OST0000: new disk, initializing [ 4719.274364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4719.279324] Lustre: 161400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4719.367865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.351820] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4727.914744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4727.921930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4728.158557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4735.381776] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4740.009848] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4742.275221] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:38:41 (1787578721) [ 5025.769169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5025.779982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5025.799225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5030.369940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5030.370291] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5030.379946] Lustre: Skipped 1 previous similar message [ 5030.400543] Lustre: Skipped 1 previous similar message [ 5031.812278] Lustre: server umount lustre-MDT0000 complete [ 5035.318942] LustreError: 159421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579016 with bad export cookie 9013572691080941139 [ 5035.323479] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5035.326267] LustreError: 159421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5035.487636] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5035.488110] LustreError: 159437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5035.509791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5035.509800] Lustre: Skipped 1 previous similar message [ 5040.611739] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5041.739708] Lustre: server umount lustre-MDT0001 complete [ 5051.818505] Lustre: server umount lustre-OST0000 complete [ 5061.750511] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 5074.233945] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5117.852822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5118.002783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5118.042977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5118.096535] Lustre: lustre-MDT0000: new disk, initializing [ 5118.189478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5118.208714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5122.925593] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5137.406725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5137.582042] Lustre: 166088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5137.764100] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5137.773680] Lustre: Skipped 1 previous similar message [ 5137.874382] Lustre: lustre-MDT0001: new disk, initializing [ 5138.065282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5138.231486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5138.252680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5144.517901] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5157.213574] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5164.292579] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5170.874238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5171.120990] Lustre: lustre-OST0000: new disk, initializing [ 5171.124731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5171.136589] Lustre: 168056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5171.216958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5176.797191] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5178.933921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5178.953875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5179.049771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5189.993994] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5196.208950] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5196.590990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5201.937492] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5202.278921] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5210.084835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.088660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5210.108374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5210.113620] Lustre: Skipped 1 previous similar message [ 5211.513293] Lustre: server umount lustre-OST0000 complete [ 5219.811614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5219.825078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5219.855989] Lustre: Skipped 1 previous similar message [ 5219.864724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5224.478897] Lustre: server umount lustre-MDT0000 complete [ 5225.442658] LustreError: 166094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.469297] LustreError: 166094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5229.976797] LustreError: 166079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579210 with bad export cookie 9013572691081099962 [ 5229.994787] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5230.001834] LustreError: 166079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5230.740727] Lustre: server umount lustre-MDT0001 complete [ 5240.930541] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:46:59 (1787579219) [ 5261.857718] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 5273.446982] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5311.154800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.335621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5311.357577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5311.454509] Lustre: lustre-MDT0000: new disk, initializing [ 5311.508375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5311.528816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5315.972106] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5325.124991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.247890] Lustre: 172825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5325.502650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5329.378394] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5338.918614] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5344.529186] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5351.507482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5351.848737] Lustre: lustre-OST0000: new disk, initializing [ 5351.851226] Lustre: Skipped 1 previous similar message [ 5351.853455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5351.855569] Lustre: Skipped 2 previous similar messages [ 5351.860483] Lustre: 174803:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5357.162599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5357.176018] Lustre: Skipped 1 previous similar message [ 5357.187556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5357.338374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5359.140414] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5368.590260] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5378.016479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5378.028585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.030770] LustreError: Skipped 1 previous similar message [ 5378.037411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5378.064173] Lustre: Skipped 2 previous similar messages [ 5378.083648] Lustre: Skipped 3 previous similar messages [ 5382.443697] Lustre: server umount lustre-OST0000 complete [ 5387.239497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5392.723712] Lustre: server umount lustre-MDT0000 complete [ 5393.378413] LustreError: 172836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5393.392593] LustreError: 172836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5396.315163] LustreError: 172816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579377 with bad export cookie 9013572691081100844 [ 5396.319101] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5396.326037] LustreError: 172816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5396.621193] Lustre: server umount lustre-MDT0001 complete [ 5405.253786] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:49:44 (1787579384) [ 5425.585611] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 5436.270098] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5470.403532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.668946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5470.696940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5470.767047] Lustre: lustre-MDT0000: new disk, initializing [ 5470.841694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5470.846652] Lustre: Skipped 2 previous similar messages [ 5470.858492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5475.677556] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5485.739433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.874734] Lustre: 179126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5486.312894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5491.544702] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5503.071766] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5509.944947] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5519.392102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5519.607579] Lustre: 181106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5526.788794] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5529.689798] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5529.766924] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5537.918085] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5550.567316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5550.580659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.586611] LustreError: Skipped 1 previous similar message [ 5550.594423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5550.612055] Lustre: Skipped 3 previous similar messages [ 5550.633711] Lustre: Skipped 4 previous similar messages [ 5556.339889] Lustre: server umount lustre-OST0000 complete [ 5568.201741] Lustre: server umount lustre-MDT0000 complete [ 5571.043749] LustreError: 180216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5571.050595] LustreError: 180216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5572.196541] LustreError: 179118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579553 with bad export cookie 9013572691081101936 [ 5572.206768] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5572.209335] LustreError: 179118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5572.424343] Lustre: server umount lustre-MDT0001 complete [ 5586.655651] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5589.415072] Key type lgssc unregistered [ 5589.630911] LNet: 182968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5589.640387] LNetError: 182968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5589.663465] LNet: Removed LNI 192.168.206.140@tcp [ 5590.344396] Key type .llcrypt unregistered [ 5590.346196] Key type ._llcrypt unregistered [ 5614.299859] Key type ._llcrypt registered [ 5614.303381] Key type .llcrypt registered [ 5614.422118] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5615.586564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5615.669798] alg: No test for adler32 (adler32-zlib) [ 5616.861249] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 5617.052924] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5618.731422] Key type lgssc registered [ 5620.236501] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5629.463299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5629.473086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5631.043944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5635.346328] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5640.168047] LustreError: 184384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.190778] LustreError: 184384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5645.282215] LustreError: 184385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.729584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5647.170317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5651.655909] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5659.457098] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5668.357866] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5677.764111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5678.307240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5685.803771] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5693.650767] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5700.843577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5722.592883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5722.601994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.618335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5726.688975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5726.696048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.727883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5726.737062] Lustre: Skipped 1 previous similar message [ 5728.696575] Lustre: server umount lustre-OST0000 complete [ 5734.368280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5734.379457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.402869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5736.935144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5736.936582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5739.512529] Lustre: server umount lustre-MDT0000 complete [ 5742.049346] LustreError: 185401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.064245] LustreError: 185401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5743.306350] LustreError: 184366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579724 with bad export cookie 1758071182393945274 [ 5743.312986] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5743.318353] LustreError: 184366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5743.612719] Lustre: server umount lustre-MDT0001 complete [ 5754.783564] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5757.774175] Key type lgssc unregistered [ 5758.119695] LNet: 187948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5758.130629] LNetError: 187948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5758.167212] LNet: Removed LNI 192.168.206.140@tcp [ 5758.931403] Key type .llcrypt unregistered [ 5758.940057] Key type ._llcrypt unregistered [ 5779.663409] Key type ._llcrypt registered [ 5779.665578] Key type .llcrypt registered [ 5779.745648] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5780.704849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5780.715057] alg: No test for adler32 (adler32-zlib) [ 5781.815752] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 5782.201627] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5783.911206] Key type lgssc registered [ 5785.098413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5792.468282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5792.492449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.919515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5797.445870] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5803.498855] LustreError: 189347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5803.517647] LustreError: 189347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5805.399584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.002450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5810.479773] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5820.127377] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5826.662749] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5833.989400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5834.311261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5838.403446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5840.885909] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5848.815295] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5858.787507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.801100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.817881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5860.802610] Lustre: lustre-OST0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 5860.835178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5860.839156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.932719] Lustre: server umount lustre-OST0000 complete [ 5871.235840] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90f309f1e800.ost_server_uuid 50 [ 5872.658531] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5887.969605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5887.993503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5888.037158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5888.047801] Lustre: Skipped 2 previous similar messages [ 5890.528694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5890.530930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5892.493705] Lustre: server umount lustre-MDT0000 complete [ 5895.655786] LustreError: 191806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.681219] LustreError: 191806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5896.424572] LustreError: 189328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579877 with bad export cookie 7252708643861897618 [ 5896.438546] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5896.721063] Lustre: server umount lustre-MDT0001 complete [ 5905.661668] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:58:04 (1787579884) [ 5912.063575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5912.615095] LustreError: 193262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5912.692895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5917.451938] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5918.181364] LustreError: 193261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5922.273484] LustreError: 193262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.013443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.572138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5931.479618] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5939.560680] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5945.269436] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5952.744563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5953.351435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5957.418489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5959.402523] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5967.808788] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5976.118810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.411945] Lustre: lustre-OST0001: new disk, initializing [ 5976.423888] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5976.446819] Lustre: 196104:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5976.600184] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5984.055866] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5986.454646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5986.461419] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5986.551988] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5993.483657] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5997.048918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5997.067780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5997.076897] Lustre: Skipped 2 previous similar messages [ 6007.282579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6007.296658] Lustre: Skipped 5 previous similar messages [ 6011.359287] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6011.440400] Lustre: server umount lustre-OST0000 complete [ 6012.388987] LustreError: 195086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6012.406028] LustreError: 195086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6017.892779] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6018.163696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6022.837940] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6023.091818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6030.209721] LustreError: 195083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6030.252787] LustreError: 195083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6037.996368] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6037.997382] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6038.002264] LustreError: Skipped 1 previous similar message [ 6038.003218] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6038.017244] Lustre: Skipped 2 previous similar messages [ 6038.030163] Lustre: Skipped 2 previous similar messages [ 6044.323302] Lustre: server umount lustre-OST0001 complete [ 6054.709976] Lustre: server umount lustre-MDT0000 complete [ 6058.997381] LustreError: 193242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580039 with bad export cookie 7252708643861898689 [ 6059.012062] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6059.013282] LustreError: 193242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6059.489369] Lustre: server umount lustre-MDT0001 complete [ 6085.232688] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:01:04 (1787580064) [ 6092.257535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.489666] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6092.524628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6092.828215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6097.468322] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6106.454695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6106.558652] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6106.597344] Lustre: 199555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6111.982488] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6120.079612] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6125.749694] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6132.356296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6132.520807] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6139.022403] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6139.553874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6146.648974] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6154.366158] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6154.440740] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6160.932629] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6168.882737] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6173.199661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6173.208732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.230058] Lustre: Skipped 3 previous similar messages [ 6173.248937] Lustre: Skipped 7 previous similar messages [ 6178.639873] Lustre: server umount lustre-OST0000 complete [ 6180.228205] LustreError: 201408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6180.250334] LustreError: 201408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6198.246254] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6201.191810] Lustre: server umount lustre-OST0001 complete [ 6208.994544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6209.019032] Lustre: Skipped 2 previous similar messages [ 6219.231798] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6219.494407] Lustre: server umount lustre-MDT0000 complete [ 6223.367948] LustreError: 202444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580204 with bad export cookie 7252708643861900089 [ 6223.371677] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6223.378467] LustreError: 202444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6223.832786] Lustre: server umount lustre-MDT0001 complete [ 6250.731732] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:03:49 (1787580229) [ 6269.235466] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 6279.605158] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6318.364246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.632151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6318.661129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6318.760512] Lustre: lustre-MDT0000: new disk, initializing [ 6318.892737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6318.905838] Lustre: Skipped 3 previous similar messages [ 6318.930416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6323.746728] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6334.545541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.675972] Lustre: 207655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6334.737780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6334.742812] Lustre: Skipped 1 previous similar message [ 6334.877657] Lustre: lustre-MDT0001: new disk, initializing [ 6335.020557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6335.034012] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6340.024973] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6349.305602] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6355.547233] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6362.183725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6362.427727] Lustre: lustre-OST0000: new disk, initializing [ 6362.431151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6362.439327] Lustre: 209622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6369.298972] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6370.406459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6370.418817] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6370.546523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6378.851995] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6384.059950] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6384.267159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6389.244612] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6389.527356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6396.388802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.398660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6396.398876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6396.398881] Lustre: Skipped 11 previous similar messages [ 6396.411429] Lustre: Skipped 2 previous similar messages [ 6396.434570] LustreError: Skipped 2 previous similar messages [ 6397.722810] Lustre: server umount lustre-OST0000 complete [ 6407.839281] Lustre: server umount lustre-MDT0000 complete [ 6411.754991] LustreError: 207661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6411.779429] LustreError: 207661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6411.801992] LustreError: 207647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580392 with bad export cookie 7252708643861901657 [ 6411.809909] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6411.817543] LustreError: 207647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6412.112042] Lustre: server umount lustre-MDT0001 complete [ 6421.869264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.795270] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6438.008473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6443.447208] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6451.156411] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6457.403619] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6464.178572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6464.494273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6464.505650] Lustre: Skipped 4 previous similar messages [ 6470.075593] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6478.617544] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6483.239954] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6483.502131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6488.282789] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6488.493806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6493.154957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6493.167227] LustreError: Skipped 1 previous similar message [ 6497.071670] Lustre: server umount lustre-OST0000 complete [ 6503.934258] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6508.636669] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6538.517469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6546.039153] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6554.259841] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6555.926712] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6558.312612] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6558.340874] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6558.341096] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6563.811520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6563.817241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6563.820946] LustreError: Skipped 1 previous similar message [ 6563.827789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6563.843846] Lustre: Skipped 5 previous similar messages [ 6563.847853] Lustre: Skipped 5 previous similar messages [ 6566.567579] Lustre: server umount lustre-OST0000 complete [ 6577.037733] Lustre: server umount lustre-MDT0000 complete [ 6579.171174] LustreError: 213082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6579.188892] LustreError: 213082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6581.026179] LustreError: 211821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580561 with bad export cookie 7252708643861902539 [ 6581.043748] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6581.046735] LustreError: 211821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6581.595781] Lustre: server umount lustre-MDT0001 complete [ 6591.429616] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:09:30 (1787580570) [ 6598.465838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6603.627677] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6612.468902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.477773] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6625.248678] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6630.975260] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6637.695435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6644.788915] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6653.073115] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6658.711255] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6658.965857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6663.906554] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6664.134732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6671.016522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6671.207980] Lustre: lustre-OST0001: new disk, initializing [ 6671.213288] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6671.240356] Lustre: 221184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6677.086538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6677.103973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6677.173269] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6677.207837] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6686.500770] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6691.038156] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6691.309073] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6696.231311] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6696.428766] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6703.076129] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6703.087802] LustreError: Skipped 2 previous similar messages [ 6704.846533] Lustre: server umount lustre-OST0001 complete [ 6711.290073] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6711.486100] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6716.010521] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6716.210605] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6746.072714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6746.530219] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6746.536895] Lustre: Skipped 5 previous similar messages [ 6751.729167] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6751.756886] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6751.759894] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6751.786808] Lustre: Skipped 2 previous similar messages [ 6754.260583] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6763.395552] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6764.888819] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6773.156513] Lustre: server umount lustre-OST0001 complete [ 6796.923037] Lustre: server umount lustre-OST0000 complete [ 6807.592020] Lustre: server umount lustre-MDT0000 complete [ 6811.593310] LustreError: 217791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580792 with bad export cookie 7252708643861903505 [ 6811.595674] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6811.606123] LustreError: 217791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6817.916837] Lustre: server umount lustre-MDT0001 complete [ 6840.632269] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:13:39 (1787580819) [ 6846.301162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6846.557534] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6846.582422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6851.173368] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6859.527765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6859.669174] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6859.685813] Lustre: 226428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6864.396498] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6871.847480] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6878.122897] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6886.363896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6886.626502] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6894.015322] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6902.293953] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6912.627832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6912.691706] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6920.476043] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6929.659827] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6931.733568] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6936.643527] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6936.889864] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6941.429364] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6941.668785] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6947.070346] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90f3086f4000.ost_server_uuid 50 [ 6948.803730] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90f3086f4000.ost_server_uuid in IDLE state after 0 sec [ 6950.184241] Lustre: Permanently deactivating lustre-OST0001 [ 6950.215144] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6954.519598] Lustre: Permanently reactivating lustre-OST0001 [ 6954.537052] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6954.541408] Lustre: Skipped 2 previous similar messages [ 6960.110782] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6960.116888] LustreError: Skipped 4 previous similar messages [ 6960.122606] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.139016] Lustre: Skipped 10 previous similar messages [ 6960.145723] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6960.151153] Lustre: Skipped 15 previous similar messages [ 6964.768256] Lustre: server umount lustre-OST0001 complete [ 6967.797799] LustreError: 228269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6967.814565] LustreError: 228269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 6975.367938] Lustre: server umount lustre-OST0000 complete [ 6985.504331] Lustre: server umount lustre-MDT0000 complete [ 6989.078402] LustreError: 226420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580969 with bad export cookie 7252708643861904793 [ 6989.081580] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6989.102142] LustreError: 226420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6989.432232] Lustre: server umount lustre-MDT0001 complete [ 7015.528154] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:16:34 (1787580994) [ 7024.839962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.219940] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7025.246777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7025.254425] Lustre: Skipped 2 previous similar messages [ 7031.073934] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7041.557552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7041.705521] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7041.766580] Lustre: 233118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7047.296859] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7056.830353] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7065.254399] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7073.967381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7074.246447] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7074.285837] Lustre: Permanently deactivating lustre-OST0000 [ 7074.318596] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7081.588408] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7082.537285] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7089.857942] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7096.821129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7096.887197] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7103.852094] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7105.247163] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7113.558714] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7116.605795] Lustre: Permanently reactivating lustre-OST0000 [ 7116.637092] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7116.649243] Lustre: Skipped 2 previous similar messages [ 7125.545106] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7125.545534] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 43 seconds. I think it's dead, and I am evicting it. exp 0000000055f3220b, cur 1787581106 deadline 1787581093 last 1787581063 [ 7125.552497] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7125.574102] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7125.624653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7138.103129] Lustre: server umount lustre-OST0001 complete [ 7156.191136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7156.474044] Lustre: server umount lustre-OST0000 complete [ 7167.187384] Lustre: server umount lustre-MDT0000 complete [ 7170.810500] LustreError: 233111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581151 with bad export cookie 7252708643861906438 [ 7170.813329] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7170.817549] LustreError: 233111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7171.364846] Lustre: server umount lustre-MDT0001 complete [ 7183.481206] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 7186.924604] Key type lgssc unregistered [ 7187.256902] LNet: 238008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7187.271985] LNetError: 238008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7187.293242] LNet: Removed LNI 192.168.206.140@tcp [ 7188.209901] Key type .llcrypt unregistered [ 7188.211551] Key type ._llcrypt unregistered [ 7216.546948] Key type ._llcrypt registered [ 7216.550572] Key type .llcrypt registered [ 7216.692638] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 7217.909072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7217.965391] alg: No test for adler32 (adler32-zlib) [ 7218.968589] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 7219.200483] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 7220.872083] Key type lgssc registered [ 7221.698242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7234.064572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7234.081079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7236.299426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7242.313672] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7246.832330] LustreError: 239476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.857502] LustreError: 239476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7251.957858] LustreError: 239475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7252.172106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.346726] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7252.403145] Lustre: 239463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7252.415362] Lustre: Permanently deactivating lustre-MDT0001 [ 7252.436859] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7252.657598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7257.028511] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7258.154131] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7265.157805] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7271.707533] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7278.739668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7279.136029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7281.193203] LustreError: 241320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7281.207427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7285.662935] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7286.241417] LustreError: 241320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7291.377541] LustreError: 241322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7295.124851] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7302.642438] LustreError: 241320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7302.667387] LustreError: 241320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7305.084689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7305.297723] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7310.344599] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7310.348447] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7315.946548] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7325.890965] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7328.919427] Lustre: Permanently reactivating lustre-MDT0001 [ 7328.936281] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7328.943955] Lustre: Skipped 2 previous similar messages [ 7332.795704] Lustre: lustre-MDT0001: Client 5b8ae742-d154-4bea-ad25-24069f6b46b1 (at 192.168.206.40@tcp) reconnecting [ 7339.122615] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.132089] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7339.138772] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7339.147880] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7340.612910] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7342.204474] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7348.782482] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7348.980287] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7354.163825] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7354.401633] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7357.830235] Lustre: Permanently deactivating lustre-MDT0001 [ 7357.858458] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7357.868455] Lustre: Skipped 2 previous similar messages [ 7365.216895] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7375.840462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7375.844343] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7375.849178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7375.866980] Lustre: Skipped 3 previous similar messages [ 7380.965759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7380.969408] Lustre: Skipped 6 previous similar messages [ 7382.130186] Lustre: server umount lustre-MDT0000 complete [ 7386.090345] LustreError: 239476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7386.101218] LustreError: 239476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7386.474587] LustreError: 240216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581367 with bad export cookie 16818224949348773896 [ 7386.487047] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7386.762214] Lustre: server umount lustre-MDT0001 complete [ 7393.784187] Lustre: server umount lustre-OST0000 complete [ 7405.917542] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:23:05 (1787581385) [ 7416.105551] Lustre: server umount lustre-OST0001 complete [ 7423.557276] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 7435.670903] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 7475.666246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.010516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7476.026232] Lustre: Skipped 2 previous similar messages [ 7476.128351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7476.269116] Lustre: lustre-MDT0000: new disk, initializing [ 7476.343059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7476.363540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7482.244297] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7494.211208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7494.377894] Lustre: 247789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7494.454697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7494.456888] Lustre: Skipped 1 previous similar message [ 7494.561418] Lustre: lustre-MDT0001: new disk, initializing [ 7494.694577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7494.724931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7494.737748] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7499.091849] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7509.795702] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7514.927751] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7523.380306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.786609] Lustre: lustre-OST0000: new disk, initializing [ 7523.797791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7523.807876] Lustre: 249759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7523.910959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7531.129394] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7533.632832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7533.650849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7533.754161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7541.607874] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7544.083138] LustreError: 247797:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7554.127117] LustreError: 247797:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7554.159272] LustreError: 247798:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7554.499314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7554.736334] Lustre: lustre-OST0001: new disk, initializing [ 7554.741507] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7554.751146] Lustre: 250876:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7562.097121] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7562.914133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7562.931981] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7563.068499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7564.217514] LustreError: 247798:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7573.835915] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7575.672732] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7578.606989] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.627510] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7578.631211] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7583.670836] Lustre: server umount lustre-OST0001 complete [ 7583.728233] LustreError: 249743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7583.744759] LustreError: 249743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7604.202317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7604.208116] Lustre: Skipped 1 previous similar message [ 7604.229773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7604.233947] Lustre: Skipped 2 previous similar messages [ 7607.556435] Lustre: server umount lustre-OST0000 complete [ 7612.393275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7612.404251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7612.409127] Lustre: Skipped 1 previous similar message [ 7612.411283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7617.761743] Lustre: server umount lustre-MDT0000 complete [ 7621.356810] LustreError: 248585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581602 with bad export cookie 16818224949348776493 [ 7621.363789] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7621.376315] LustreError: 248585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7627.749988] Lustre: server umount lustre-MDT0001 complete [ 7637.319848] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 7640.342667] Key type lgssc unregistered [ 7640.711667] LNet: 252911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7640.723990] LNetError: 252911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7641.766170] LNet: Removed LNI 192.168.206.140@tcp [ 7642.737208] Key type .llcrypt unregistered [ 7642.742863] Key type ._llcrypt unregistered [ 7687.877993] Key type ._llcrypt registered [ 7687.882397] Key type .llcrypt registered [ 7687.986382] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 7689.197350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7689.230262] alg: No test for adler32 (adler32-zlib) [ 7690.244742] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 7690.476726] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 7692.135169] Key type lgssc registered [ 7693.116105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7700.826373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7700.853745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7702.168844] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7702.198476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7702.425525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7706.841984] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7715.414455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7715.585076] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7715.617747] Lustre: 255119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7715.901649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7720.204980] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7728.954936] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7735.242478] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7742.935334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7743.228284] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7743.451677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7750.430754] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7751.292856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7758.320738] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7812.576272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7812.578977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.601641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7813.103584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7813.106399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7813.118892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7813.135388] Lustre: Skipped 1 previous similar message [ 7817.879878] Lustre: server umount lustre-OST0000 complete [ 7822.798413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7835.980498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7836.350677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7838.376570] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7838.404764] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7843.607327] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7851.449838] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7853.749086] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7853.762847] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7895.008451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7895.029019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7895.031929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7895.041795] LustreError: Skipped 1 previous similar message [ 7895.049134] Lustre: Skipped 1 previous similar message [ 7897.101397] Lustre: server umount lustre-OST0000 complete [ 7905.248562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7905.270923] Lustre: Skipped 1 previous similar message [ 7905.293750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7905.313179] Lustre: Skipped 1 previous similar message [ 7908.943770] Lustre: server umount lustre-MDT0000 complete [ 7913.737401] LustreError: 255112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581894 with bad export cookie 12185465084599421372 [ 7913.748757] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7913.760918] LustreError: 255112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7914.411832] Lustre: server umount lustre-MDT0001 complete [ 7928.417092] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 7930.661154] Key type lgssc unregistered [ 7930.999894] LNet: 260550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7931.005735] LNetError: 260550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7932.071547] LNet: Removed LNI 192.168.206.140@tcp [ 7932.983397] Key type .llcrypt unregistered [ 7932.996729] Key type ._llcrypt unregistered [ 7959.528467] Key type ._llcrypt registered [ 7959.531399] Key type .llcrypt registered [ 7959.628256] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 7960.429911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7960.604705] alg: No test for adler32 (adler32-zlib) [ 7961.876910] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 7962.075767] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 7963.759148] Key type lgssc registered [ 7964.743918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7972.691476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7972.717337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7974.213926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7979.052897] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7984.610645] LustreError: 261967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7984.631099] LustreError: 261967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7986.848582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.172640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7991.427444] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7998.961150] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8004.720651] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8012.347797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8012.748160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.968321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8020.292652] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8028.970682] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8055.778072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8055.790334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8055.821547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8056.801020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8056.806687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8056.812802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.850307] Lustre: Skipped 1 previous similar message [ 8059.874103] Lustre: server umount lustre-OST0000 complete [ 8063.975202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8063.982763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8063.994863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8067.047989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8067.052496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8072.163962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8072.180064] Lustre: Skipped 2 previous similar messages [ 8078.303387] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8078.663418] Lustre: server umount lustre-MDT0000 complete [ 8081.379603] LustreError: 262695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8081.415472] LustreError: 262695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8082.954535] LustreError: 261950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582063 with bad export cookie 18426587454077082530 [ 8082.964210] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8083.603902] Lustre: server umount lustre-MDT0001 complete [ 8096.684929] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8100.725696] Key type lgssc unregistered [ 8101.117801] LNet: 266279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8101.139234] LNetError: 266279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8101.162137] LNet: Removed LNI 192.168.206.140@tcp [ 8102.285143] Key type .llcrypt unregistered [ 8102.286762] Key type ._llcrypt unregistered [ 8123.080907] Key type ._llcrypt registered [ 8123.083103] Key type .llcrypt registered [ 8123.226436] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8125.089332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8125.156558] alg: No test for adler32 (adler32-zlib) [ 8126.384503] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 8126.754752] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 8128.423258] Key type lgssc registered [ 8129.819450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8138.840057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8138.860064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.543025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8146.135358] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8151.021787] LustreError: 267502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8151.039154] LustreError: 267502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8156.131735] LustreError: 267501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8156.211461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8156.835202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8162.436518] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8170.968400] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8178.060135] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8185.656425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8185.786822] LustreError: 269320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8186.060573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8191.483200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8192.845890] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8201.233576] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8217.056662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8217.068830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8217.081520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8217.094126] Lustre: Skipped 1 previous similar message [ 8219.888893] Lustre: server umount lustre-OST0000 complete [ 8227.302543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8227.320087] Lustre: Skipped 1 previous similar message [ 8227.322810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8227.327290] Lustre: Skipped 1 previous similar message [ 8230.519416] Lustre: server umount lustre-MDT0000 complete [ 8232.427866] LustreError: 267502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8232.449296] LustreError: 267502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8235.478260] LustreError: 267480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582216 with bad export cookie 15593992895630607885 [ 8235.497971] LustreError: 267480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8235.503090] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8235.943862] Lustre: server umount lustre-MDT0001 complete [ 8248.828653] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8252.088781] Key type lgssc unregistered [ 8252.425465] LNet: 271186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8252.443273] LNetError: 271186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8252.462481] LNet: Removed LNI 192.168.206.140@tcp [ 8253.521609] Key type .llcrypt unregistered [ 8253.523659] Key type ._llcrypt unregistered [ 8282.457993] Key type ._llcrypt registered [ 8282.463251] Key type .llcrypt registered [ 8282.550224] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8283.628378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8283.663388] alg: No test for adler32 (adler32-zlib) [ 8284.797973] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 8285.166858] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 8286.887451] Key type lgssc registered [ 8288.132709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8296.787533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8296.847115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8298.365259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8302.179452] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8308.712417] LustreError: 272622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8308.732640] LustreError: 272622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8310.093543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8310.477210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8314.686773] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8323.434268] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8329.351797] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8335.384636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8335.656659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8343.871050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8343.993899] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8352.435118] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8381.925057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8381.933109] Lustre: Skipped 1 previous similar message [ 8381.933586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8381.938101] Lustre: Skipped 1 previous similar message [ 8387.012614] Lustre: server umount lustre-OST0000 complete [ 8392.163523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8392.169412] Lustre: Skipped 1 previous similar message [ 8392.171375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8392.179541] Lustre: Skipped 1 previous similar message [ 8397.211152] Lustre: server umount lustre-MDT0000 complete [ 8397.288099] LustreError: 272617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8397.335199] LustreError: 272617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8401.393747] LustreError: 272621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8401.426091] LustreError: 272621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8402.281346] LustreError: 272601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582383 with bad export cookie 18390330865352758748 [ 8402.283032] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8402.289108] LustreError: 272601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8402.543988] Lustre: server umount lustre-MDT0001 complete [ 8415.689884] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8418.041308] Key type lgssc unregistered [ 8418.244416] LNet: 276930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8418.253658] LNetError: 276930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8418.264774] LNet: Removed LNI 192.168.206.140@tcp [ 8418.847639] Key type .llcrypt unregistered [ 8418.850766] Key type ._llcrypt unregistered [ 8435.930333] Key type ._llcrypt registered [ 8435.932317] Key type .llcrypt registered [ 8436.001829] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8436.903695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8436.960525] alg: No test for adler32 (adler32-zlib) [ 8437.939386] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 8438.098651] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 8439.735402] Key type lgssc registered [ 8440.610425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8448.476694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8448.497871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8450.010378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8455.445275] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8459.235433] LustreError: 278166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8459.244587] LustreError: 278166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8464.361086] LustreError: 278168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8466.479334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.985536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8471.595122] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8478.796332] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8487.669878] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8494.701781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8495.043102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8501.122559] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8509.181606] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8519.619431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8535.009550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8535.018333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8535.023537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8535.054478] Lustre: Skipped 1 previous similar message [ 8539.724812] Lustre: server umount lustre-OST0000 complete [ 8545.250718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8545.274446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8545.292651] Lustre: Skipped 2 previous similar messages [ 8550.369859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8550.384494] Lustre: Skipped 1 previous similar message [ 8550.811313] Lustre: server umount lustre-MDT0000 complete [ 8555.280257] LustreError: 278216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582536 with bad export cookie 7591076696244861224 [ 8555.293401] LustreError: 278216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8555.295548] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8555.731258] Lustre: server umount lustre-MDT0001 complete [ 8568.559435] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8571.744644] Key type lgssc unregistered [ 8572.296530] LNet: 281895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8572.302825] LNetError: 281895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8572.317890] LNet: Removed LNI 192.168.206.140@tcp [ 8573.262126] Key type .llcrypt unregistered [ 8573.277451] Key type ._llcrypt unregistered [ 8590.419703] Key type ._llcrypt registered [ 8590.421590] Key type .llcrypt registered [ 8590.570608] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8611.053797] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 8626.126565] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8627.603783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8627.626833] alg: No test for adler32 (adler32-zlib) [ 8628.799913] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 8629.079405] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 8630.855252] Key type lgssc registered [ 8632.100770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8665.755255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8665.796943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.029549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8667.053563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8667.116768] Lustre: lustre-MDT0000: new disk, initializing [ 8667.197449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8667.211780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8671.989268] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8683.226785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8683.379875] Lustre: 285451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8683.427850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8683.431538] Lustre: Skipped 1 previous similar message [ 8683.532069] Lustre: lustre-MDT0001: new disk, initializing [ 8683.676386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8683.716546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8683.731530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8688.674268] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8699.754725] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8706.088666] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8713.553991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8713.794371] Lustre: lustre-OST0000: new disk, initializing [ 8713.800424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8713.811863] Lustre: 287422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8713.934425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8720.496380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8720.507232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8720.682220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8721.010546] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8731.441874] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8735.471440] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8735.602879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8739.245960] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8739.474418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8741.354326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8741.364757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8741.369561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8741.372538] LustreError: Skipped 1 previous similar message [ 8741.379874] Lustre: Skipped 1 previous similar message [ 8746.497878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8746.510798] Lustre: Skipped 1 previous similar message [ 8747.646590] Lustre: server umount lustre-OST0000 complete [ 8755.689261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8755.705564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8755.732113] Lustre: Skipped 1 previous similar message [ 8755.743221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8756.707334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8759.268804] Lustre: server umount lustre-MDT0000 complete [ 8764.199074] LustreError: 285442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582745 with bad export cookie 5331233310246356007 [ 8764.216810] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8764.479490] Lustre: server umount lustre-MDT0001 complete [ 8773.252701] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:45:52 (1787582752) [ 8780.626291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8781.251406] LustreError: 289834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8781.265989] LustreError: 289834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8781.337497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8785.577963] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8786.402278] LustreError: 289835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.527439] LustreError: 289834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8794.378081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8794.760827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8799.827680] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8807.743509] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8814.639626] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8822.573453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8823.052567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8829.387431] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8837.313502] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8849.889574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8849.904219] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.908805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8849.913509] Lustre: Skipped 1 previous similar message [ 8849.920447] Lustre: Skipped 2 previous similar messages [ 8854.156991] Lustre: server umount lustre-OST0000 complete [ 8860.149344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.157442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8860.200187] Lustre: Skipped 1 previous similar message [ 8860.205425] Lustre: Skipped 1 previous similar message [ 8865.258521] LustreError: 291679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8865.289794] LustreError: 291679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8866.940996] Lustre: server umount lustre-MDT0000 complete [ 8869.349550] LustreError: 289834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8869.384322] LustreError: 289834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8871.385555] LustreError: 289816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582852 with bad export cookie 5331233310246356882 [ 8871.402910] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8871.407334] LustreError: 289816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8872.464424] Lustre: server umount lustre-MDT0001 complete [ 8885.698620] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8887.993961] Key type lgssc unregistered [ 8888.287332] LNet: 293373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8888.317309] LNetError: 293373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8889.382075] LNet: Removed LNI 192.168.206.140@tcp [ 8890.271379] Key type .llcrypt unregistered [ 8890.275945] Key type ._llcrypt unregistered [ 8917.620908] Key type ._llcrypt registered [ 8917.623737] Key type .llcrypt registered [ 8917.757980] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 8933.641679] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8934.792960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8934.799892] alg: No test for adler32 (adler32-zlib) [ 8935.878642] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 8936.086909] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 8937.759206] Key type lgssc registered [ 8938.897816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8988.759742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8988.781724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8990.054285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8990.093729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8990.184808] Lustre: lustre-MDT0000: new disk, initializing [ 8990.261943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8990.279505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8994.414427] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9005.270691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9005.419986] Lustre: 297203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9005.484826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9005.492242] Lustre: Skipped 1 previous similar message [ 9005.610298] Lustre: lustre-MDT0001: new disk, initializing [ 9005.733117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9005.799463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9005.820163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9010.423786] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9019.899822] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9025.592465] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9032.047869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9032.316721] Lustre: lustre-OST2710: new disk, initializing [ 9032.320509] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9032.337390] Lustre: 299171:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9032.423300] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9037.855584] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9040.968154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9040.980550] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9041.098818] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9046.880919] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9053.573378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9053.700489] Lustre: lustre-OST03e8: new disk, initializing [ 9053.705457] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9053.711479] Lustre: 300235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9053.794245] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9059.422982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9059.435887] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9059.574196] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9060.246151] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9068.148736] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9074.962770] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9075.247561] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9079.801775] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9080.022936] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9084.265979] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9084.409620] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9089.089736] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9089.290785] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9095.658854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9095.682265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9095.691910] Lustre: Skipped 2 previous similar messages [ 9095.695695] Lustre: Skipped 1 previous similar message [ 9096.341756] Lustre: server umount lustre-MDT0000 complete [ 9101.054859] LustreError: 297196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583081 with bad export cookie 5316294850605315206 [ 9101.063228] LustreError: 297196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9101.065397] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9101.413975] Lustre: server umount lustre-MDT0001 complete [ 9116.214327] Lustre: server umount lustre-OST2710 complete [ 9130.116456] Lustre: server umount lustre-OST03e8 complete [ 9147.736543] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 9156.933070] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 9191.050403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.247472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9191.266241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9191.337384] Lustre: lustre-MDT0000: new disk, initializing [ 9191.415852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9191.437921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9195.721979] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9206.211311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9206.319464] Lustre: 305480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9206.378601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9206.388397] Lustre: Skipped 1 previous similar message [ 9206.464307] Lustre: lustre-MDT0001: new disk, initializing [ 9206.579925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9206.604566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9206.612805] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9211.536380] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9221.022973] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9227.069606] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9234.016140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9234.246461] Lustre: lustre-OST0000: new disk, initializing [ 9234.253393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9234.263410] Lustre: 307449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9234.368447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9241.215409] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9241.643118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9241.656800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9241.745771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9251.585972] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9256.678972] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9256.879884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9260.869120] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9261.082795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9267.683353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9267.694605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9267.696972] Lustre: Skipped 2 previous similar messages [ 9267.700612] Lustre: Skipped 2 previous similar messages [ 9269.195899] Lustre: server umount lustre-OST0000 complete [ 9272.803811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9272.809445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9272.817407] Lustre: Skipped 1 previous similar message [ 9272.837303] Lustre: Skipped 1 previous similar message [ 9277.936316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9277.941467] Lustre: Skipped 1 previous similar message [ 9278.582821] Lustre: server umount lustre-MDT0000 complete [ 9282.379578] LustreError: 305475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583263 with bad export cookie 5316294850605316886 [ 9282.385908] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9282.405314] LustreError: 305475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9282.857192] Lustre: server umount lustre-MDT0001 complete [ 9291.552625] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:54:30 (1787583270) [ 9292.964550] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9294.472891] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:54:33 (1787583273) [ 9297.991858] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 9325.149377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9325.429634] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9325.465794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9325.615341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9330.492513] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9338.153633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9338.298533] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9338.338796] Lustre: 311305:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9342.916494] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9366.512021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9366.887499] Lustre: Denying initial registration attempt from nid 192.168.206.140@tcp, specified as failover [ 9366.909618] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9366.927803] LustreError: 312762:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9366.934520] LustreError: 312762:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9366.944640] LustreError: 312762:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9367.082673] Lustre: server umount lustre-OST0000 complete [ 9367.086530] LustreError: 312762:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9374.688273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9374.700712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9374.709680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9381.031330] Lustre: server umount lustre-MDT0000 complete [ 9382.881852] LustreError: 311315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9382.893839] LustreError: 311315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9385.268175] LustreError: 311299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583366 with bad export cookie 5316294850605317768 [ 9385.270509] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9385.292426] LustreError: 311299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9385.843374] Lustre: server umount lustre-MDT0001 complete [ 9397.528988] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 9400.530905] Key type lgssc unregistered [ 9400.960744] LNet: 313886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9400.969686] LNetError: 313886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9400.996488] LNet: Removed LNI 192.168.206.140@tcp [ 9401.851332] Key type .llcrypt unregistered [ 9401.854193] Key type ._llcrypt unregistered [ 9414.484772] Key type ._llcrypt registered [ 9414.489350] Key type .llcrypt registered [ 9414.664701] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 9415.932301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9416.138855] alg: No test for adler32 (adler32-zlib) [ 9417.288516] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 9417.627798] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 9419.383229] Key type lgssc registered [ 9420.534906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9468.891341] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 9478.834059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9478.849994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9480.187391] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9480.228470] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9480.418722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9485.453562] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9495.197537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9495.401810] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9495.444120] Lustre: 316667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9495.729588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9500.903461] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9510.074457] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9517.152399] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9537.398744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9537.709706] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9538.024119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.933460] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9552.718515] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9558.505955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9558.509284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9558.510049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9558.517352] LustreError: Skipped 1 previous similar message [ 9558.535687] Lustre: Skipped 1 previous similar message [ 9563.626236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9563.636855] Lustre: Skipped 1 previous similar message [ 9568.740543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9568.744681] Lustre: Skipped 1 previous similar message [ 9569.247173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9569.337490] Lustre: server umount lustre-OST0000 complete [ 9572.835161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9572.844223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.870451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9572.883026] Lustre: Skipped 1 previous similar message [ 9573.857317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9578.908757] Lustre: server umount lustre-MDT0000 complete [ 9578.976279] LustreError: 316672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9578.988610] LustreError: 316672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9582.391123] LustreError: 316659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583563 with bad export cookie 2120376578589156950 [ 9582.402417] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9582.404561] LustreError: 316659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9582.691709] Lustre: server umount lustre-MDT0001 complete [ 9593.908472] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 9596.709321] Key type lgssc unregistered [ 9597.077985] LNet: 320248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9597.091215] LNetError: 320248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9597.129259] LNet: Removed LNI 192.168.206.140@tcp [ 9597.915900] Key type .llcrypt unregistered [ 9597.920928] Key type ._llcrypt unregistered [ 9622.137922] Key type ._llcrypt registered [ 9622.141548] Key type .llcrypt registered [ 9622.284409] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 9623.598162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9623.748129] alg: No test for adler32 (adler32-zlib) [ 9624.915780] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 9625.198863] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 9626.879269] Key type lgssc registered [ 9628.210847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9636.277916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9636.299733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9637.687786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9642.231594] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9648.105963] LustreError: 321664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9648.132206] LustreError: 321664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9651.783244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9652.198551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9657.240986] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9664.826517] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9670.517026] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9679.788028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9680.166538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9686.667887] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9695.628273] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9708.002536] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9708.008147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9708.033422] Lustre: Skipped 1 previous similar message [ 9708.982792] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 9708.998835] Lustre: Skipped 2 previous similar messages [ 9710.912256] Lustre: server umount lustre-MDT0000 complete [ 9713.123675] LustreError: 321660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9713.145993] LustreError: 321660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9714.731818] LustreError: 321644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583695 with bad export cookie 13927875402524086753 [ 9714.734289] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9714.742661] LustreError: 321644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9718.240753] LustreError: 321660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.244873] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9718.259628] LustreError: 321660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9718.281089] Lustre: Skipped 1 previous similar message [ 9718.285179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9721.157142] Lustre: server umount lustre-MDT0001 complete [ 9726.994208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9736.019052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9740.096952] LustreError: 323495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9740.112378] LustreError: 323495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9740.174381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9740.232610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9744.291202] LustreError: 323496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9744.313758] LustreError: 323496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9745.403627] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9749.432181] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9754.596118] LustreError: 325272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9754.622414] LustreError: 325272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9756.010190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9756.552815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9761.546485] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9761.783804] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9770.079155] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9772.104776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9778.823716] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9780.932939] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9786.336771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9786.341876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.351039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9786.356057] Lustre: Skipped 2 previous similar messages [ 9787.360103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9792.439483] Lustre: server umount lustre-OST0000 complete [ 9797.615859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9797.624588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9797.640335] Lustre: Skipped 1 previous similar message [ 9797.648300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9797.655343] Lustre: Skipped 3 previous similar messages [ 9801.848979] Lustre: server umount lustre-MDT0000 complete [ 9802.721168] LustreError: 325277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9802.737973] LustreError: 325277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9805.235290] LustreError: 321643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583786 with bad export cookie 13927875402524094383 [ 9805.238199] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9805.245686] LustreError: 321643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9805.496694] Lustre: server umount lustre-MDT0001 complete [ 9816.322464] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 9818.579848] Key type lgssc unregistered [ 9818.815488] LNet: 327949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9818.827712] LNetError: 327949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9818.841290] LNet: Removed LNI 192.168.206.140@tcp [ 9819.848766] Key type .llcrypt unregistered [ 9819.850858] Key type ._llcrypt unregistered [ 9848.109906] Key type ._llcrypt registered [ 9848.116634] Key type .llcrypt registered [ 9848.298373] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 9849.391428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9849.556601] alg: No test for adler32 (adler32-zlib) [ 9850.674318] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [ 9850.938842] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 9852.601371] Key type lgssc registered [ 9854.040978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9862.651920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9862.667046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9864.064385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9869.621486] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9873.388524] LustreError: 329368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9873.409128] LustreError: 329368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9878.497739] LustreError: 329367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9880.028363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9880.437618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9885.085922] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9894.197699] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9900.870182] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9906.144230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9906.152685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9906.168606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9910.240384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9910.242840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9911.595811] Lustre: server umount lustre-MDT0000 complete [ 9915.363910] LustreError: 329364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9915.394179] LustreError: 329364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9915.502776] LustreError: 329348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583896 with bad export cookie 11796611669393387338 [ 9915.519128] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9915.817949] Lustre: server umount lustre-MDT0001 complete [ 9931.125243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9931.400428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9931.456844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9931.667483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9935.579284] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9943.745531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.848501] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9943.887852] Lustre: 331953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9944.199677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9948.620739] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9955.291294] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9960.060665] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9968.814242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9969.044826] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9969.055506] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9969.063255] LustreError: 333839:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9969.068327] LustreError: 333839:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9969.072401] LustreError: 333839:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9969.197469] Lustre: server umount lustre-OST0000 complete [ 9969.205980] LustreError: 333839:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9979.243085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9979.444831] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9979.606191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9985.325307] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 9985.674093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9992.765375] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10000.769093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10000.884575] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10000.992167] Lustre: lustre-OST0001: new disk, initializing [10001.001619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10001.014653] Lustre: 335380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10001.131795] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10007.504786] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10009.685041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10009.692456] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10009.791962] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10014.507412] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10020.320513] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10020.330319] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10020.333634] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10020.344813] Lustre: Skipped 1 previous similar message [10020.346739] Lustre: Skipped 1 previous similar message [10023.491434] Lustre: server umount lustre-OST0001 complete [10025.449137] LustreError: 334304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10025.461282] LustreError: 334304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10030.571257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10030.577665] LustreError: 334306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10030.580534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10030.580546] Lustre: Skipped 1 previous similar message [10030.584116] Lustre: Skipped 1 previous similar message [10030.597802] LustreError: 334306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10035.701658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10035.716194] Lustre: Skipped 2 previous similar messages [10040.807150] LustreError: 334307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10040.839186] LustreError: 334307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10041.314459] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10041.762305] Lustre: server umount lustre-OST0000 complete [10045.923970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10045.930580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10045.932635] Lustre: Skipped 1 previous similar message [10045.943079] Lustre: Skipped 3 previous similar messages [10060.255186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10060.428923] Lustre: server umount lustre-MDT0000 complete [10061.280543] LustreError: 332841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10061.299022] LustreError: 332841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10063.984420] LustreError: 332695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584044 with bad export cookie 11796611669393387891 [10063.985299] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10063.996476] LustreError: 332695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10064.423112] Lustre: server umount lustre-MDT0001 complete [10074.996253] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [10077.370953] Key type lgssc unregistered [10077.665667] LNet: 337346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10077.674381] LNetError: 337346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10077.689607] LNet: Removed LNI 192.168.206.140@tcp [10078.516104] Key type .llcrypt unregistered [10078.517646] Key type ._llcrypt unregistered [10147.135697] Key type ._llcrypt registered [10147.137173] Key type .llcrypt registered [10147.209899] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [10164.238596] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [10165.441711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10165.512681] alg: No test for adler32 (adler32-zlib) [10166.505445] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [10166.693761] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [10168.335140] Key type lgssc registered [10169.249392] Lustre: Echo OBD driver; http://www.lustre.org/ [10199.051699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10199.063972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10200.317362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10200.338586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10200.406056] Lustre: lustre-MDT0000: new disk, initializing [10200.467843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10200.483331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10204.370363] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10213.928580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10214.026707] Lustre: 342281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10214.068530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10214.072258] Lustre: Skipped 1 previous similar message [10214.119846] Lustre: lustre-MDT0001: new disk, initializing [10214.225317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10214.257060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10214.269244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10217.482756] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10226.504332] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10233.907412] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10240.644793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10240.880456] Lustre: lustre-OST0000: new disk, initializing [10240.885793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10240.892538] Lustre: 344249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10240.969326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10246.377525] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10247.747988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10247.757495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10247.870213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10255.915706] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10260.614983] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10260.833467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10264.792760] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10265.012571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10268.647586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10268.649128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10268.650388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10268.658543] Lustre: Skipped 1 previous similar message [10273.311630] Lustre: server umount lustre-OST0000 complete [10278.880555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10278.906302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10278.916208] Lustre: Skipped 2 previous similar messages [10283.735937] Lustre: server umount lustre-MDT0000 complete [10287.905089] LustreError: 343083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584268 with bad export cookie 17493078230028440356 [10287.921367] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10287.931650] LustreError: 343083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10288.332590] Lustre: server umount lustre-MDT0001 complete [10299.324834] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:11:18 (1787584278) [10306.177796] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:11:25 (1787584285) [10315.793959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10316.237978] LustreError: 347004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10316.259692] LustreError: 347004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10316.334946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10321.379397] LustreError: 347005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10324.464861] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10326.500299] LustreError: 347004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10331.616196] LustreError: 347005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10336.740849] LustreError: 347004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10339.341506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10340.397588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10347.756815] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10355.997403] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10362.369196] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10368.470271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10368.936050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10375.678479] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10383.678473] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10397.958414] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10399.690539] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10401.535637] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10403.663962] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10405.738709] Lustre: Failing over lustre-MDT0000 [10406.225207] Lustre: server umount lustre-MDT0000 complete [10406.830421] LustreError: 346999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.880541] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10406.898677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10406.906178] Lustre: Skipped 3 previous similar messages [10406.918721] LustreError: Skipped 1 previous similar message [10422.241861] Lustre: 340507:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584387/real 1787584387] req@00000000cf5cee21 x1874417834597376/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787584403 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10422.276048] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10423.097887] LustreError: 346999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10423.124477] LustreError: 346999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10425.279775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.490552] LustreError: 340503:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002edfed79 x1874417834603648/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10432.797423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10432.870121] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10433.406682] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10437.306416] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10438.139403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10438.191933] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10438.241398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10447.575745] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10449.325923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10452.903647] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10454.971794] Lustre: Failing over lustre-MDT0000 [10455.232166] Lustre: server umount lustre-MDT0000 complete [10458.595924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10458.596079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10458.597147] LustreError: 346998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10458.597154] LustreError: 346998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10458.621919] Lustre: Skipped 2 previous similar messages [10474.155743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10474.246657] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10474.370086] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (not set up) [10474.578685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10474.650788] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10479.496854] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10479.604971] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10479.618441] Lustre: Skipped 2 previous similar messages [10479.648538] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10479.769325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10480.178507] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10489.863549] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10491.502557] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10500.089190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10500.092942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10500.115426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10504.467448] Lustre: server umount lustre-MDT0000 complete [10508.788218] LustreError: 347741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584489 with bad export cookie 17493078230028443065 [10508.795930] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10508.801106] LustreError: 347741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10509.094580] Lustre: server umount lustre-MDT0001 complete [10522.478848] Lustre: server umount lustre-OST0000 complete [10538.850746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10539.341578] LustreError: 353369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10539.362289] LustreError: 353369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [10539.432196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10544.273853] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10552.668909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10557.265589] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10565.569321] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10571.473061] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10578.603322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10585.003511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10585.474883] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10593.400628] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10605.536121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10605.542227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10605.550254] Lustre: Skipped 2 previous similar messages [10605.553723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10605.557461] Lustre: Skipped 2 previous similar messages [10617.828397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10617.835393] Lustre: Skipped 4 previous similar messages [10618.335139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10618.524536] Lustre: server umount lustre-OST0000 complete [10622.944087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10622.959679] Lustre: Skipped 2 previous similar messages [10628.564914] Lustre: server umount lustre-MDT0000 complete [10632.742963] LustreError: 353351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584613 with bad export cookie 17493078230028443814 [10632.744626] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10632.759596] LustreError: 353351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10633.202385] Lustre: server umount lustre-MDT0001 complete [10643.675775] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [10646.197503] Key type lgssc unregistered [10646.492928] LNet: 356905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10646.504375] LNetError: 356905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10647.530489] LNet: Removed LNI 192.168.206.140@tcp [10648.487681] Key type .llcrypt unregistered [10648.492285] Key type ._llcrypt unregistered [10672.013373] Key type ._llcrypt registered [10672.016271] Key type .llcrypt registered [10672.122343] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [10672.966650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10673.114653] alg: No test for adler32 (adler32-zlib) [10674.236420] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [10674.495089] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [10676.183463] Key type lgssc registered [10677.282207] Lustre: Echo OBD driver; http://www.lustre.org/ [10684.922696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10684.937772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.351086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10690.819695] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10696.677319] LustreError: 358402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10696.694991] LustreError: 358402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10699.631828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10700.066350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10704.288419] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10712.075106] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10719.048888] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10726.395149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10726.763741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10731.894803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10732.863535] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10740.332322] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10756.578611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10756.595207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10756.619433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10759.654642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10759.660513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10759.676280] Lustre: Skipped 1 previous similar message [10761.576937] Lustre: server umount lustre-MDT0000 complete [10764.772221] LustreError: 358402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10764.796266] LustreError: 358402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10765.928813] LustreError: 358383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584746 with bad export cookie 12625663525035480712 [10765.974557] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10766.408514] Lustre: server umount lustre-MDT0001 complete [10782.010436] Lustre: server umount lustre-OST0000 complete [10795.388868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10795.834202] LustreError: 362134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10795.931373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10800.713958] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10801.121627] LustreError: 362135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10805.221938] LustreError: 362134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.767599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10809.107287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10813.294303] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10821.040861] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10826.520705] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10833.341555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10833.745579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10839.837444] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10840.948737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10847.216036] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10860.512607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10860.520069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10860.527179] Lustre: Skipped 1 previous similar message [10860.530910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10864.088171] Lustre: server umount lustre-MDT0000 complete [10866.657166] LustreError: 362134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.676063] LustreError: 362134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10867.028798] LustreError: 362115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584847 with bad export cookie 12625663525035481825 [10867.033418] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10867.042685] LustreError: 362115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10867.243746] Lustre: server umount lustre-MDT0001 complete [10880.913784] Lustre: server umount lustre-OST0000 complete [10896.512648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10897.001720] LustreError: 366046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10897.130343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10900.947318] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10909.141635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.989804] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10922.068970] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10928.015507] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10934.918680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10935.227667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10935.231377] Lustre: Skipped 1 previous similar message [10941.363370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10941.695556] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [10949.042887] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10955.108188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10955.108585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10955.138848] Lustre: Skipped 2 previous similar messages [10955.158268] Lustre: Skipped 4 previous similar messages [10956.427200] Lustre: server umount lustre-OST0000 complete [10960.865669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10960.870915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10960.879971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10970.592611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10970.599434] Lustre: Skipped 3 previous similar messages [10975.199943] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10975.411684] Lustre: server umount lustre-MDT0000 complete [10975.716711] LustreError: 368537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10975.733074] LustreError: 368537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10979.316538] LustreError: 367888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584960 with bad export cookie 12625663525035482847 [10979.317898] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10979.329460] LustreError: 367888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10979.986838] Lustre: server umount lustre-MDT0001 complete [10991.682205] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [10994.329861] Key type lgssc unregistered [10994.599558] LNet: 369583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10994.608738] LNetError: 369583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10994.624650] LNet: Removed LNI 192.168.206.140@tcp [10995.202855] Key type .llcrypt unregistered [10995.205557] Key type ._llcrypt unregistered [11021.176731] Key type ._llcrypt registered [11021.183432] Key type .llcrypt registered [11021.289668] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [11022.109580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11022.195903] alg: No test for adler32 (adler32-zlib) [11023.277569] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [11023.506366] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [11025.159119] Key type lgssc registered [11026.142758] Lustre: Echo OBD driver; http://www.lustre.org/ [11033.052251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11033.071601] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11033.078910] LustreError: 371028:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11033.092770] LustreError: 371028:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11033.097161] LustreError: 371028:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11033.102357] LustreError: 371028:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11033.107668] LustreError: 371028:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11040.778644] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11040.784135] LustreError: 371425:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11040.803414] LustreError: 371425:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11040.808970] LustreError: 371425:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11040.823285] LustreError: 371425:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11040.836808] LustreError: 371425:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11061.046791] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [11064.130297] Key type lgssc unregistered [11064.513165] LNet: 372432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11064.520943] LNetError: 372432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11064.539787] LNet: Removed LNI 192.168.206.140@tcp [11065.589706] Key type .llcrypt unregistered [11065.591934] Key type ._llcrypt unregistered [11085.376173] Key type ._llcrypt registered [11085.380196] Key type .llcrypt registered [11085.533606] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [11101.985624] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [11102.781725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11102.843084] alg: No test for adler32 (adler32-zlib) [11104.091534] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [11104.431883] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [11106.184072] Key type lgssc registered [11107.491533] Lustre: Echo OBD driver; http://www.lustre.org/ [11143.268545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11143.311913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11144.598630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11144.652442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11144.727809] Lustre: lustre-MDT0000: new disk, initializing [11144.801919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11144.824169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11149.847980] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11160.732932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11160.831938] Lustre: 375588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11160.887315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11160.889607] Lustre: Skipped 1 previous similar message [11160.944865] Lustre: lustre-MDT0001: new disk, initializing [11161.031955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11161.068407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11161.079669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11165.880693] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11176.180406] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11182.380215] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11190.343490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11190.666822] Lustre: lustre-OST0000: new disk, initializing [11190.676438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11190.686477] Lustre: 377559:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11190.825117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11195.965831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11195.979626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11196.177041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11198.218974] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11207.265894] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11211.676122] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11211.874152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11216.618523] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11216.843776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11221.991463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11221.992043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11222.007577] LustreError: Skipped 1 previous similar message [11222.011829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11222.068122] Lustre: Skipped 1 previous similar message [11227.116046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11227.135743] Lustre: Skipped 2 previous similar messages [11232.226768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11232.235066] Lustre: Skipped 1 previous similar message [11233.247090] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11233.406676] Lustre: server umount lustre-OST0000 complete [11237.349804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.365351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11237.366660] Lustre: Skipped 1 previous similar message [11237.388544] Lustre: Skipped 1 previous similar message [11242.469048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11242.479339] Lustre: Skipped 1 previous similar message [11243.620179] Lustre: server umount lustre-MDT0000 complete [11247.455389] LustreError: 375582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585228 with bad export cookie 15674877389055921980 [11247.462205] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11247.470382] LustreError: 375582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11247.852478] Lustre: server umount lustre-MDT0001 complete [11256.689977] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11258.608335] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:27:17 (1787585237) [11265.116393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11265.594173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11270.364699] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11270.631464] LustreError: 380022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11270.651902] LustreError: 380022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11275.747940] LustreError: 380021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.439246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11278.884270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11282.892714] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11289.971672] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11295.237675] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11301.302459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11301.583297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11307.776971] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11315.759380] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11322.208689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11322.349390] Lustre: lustre-OST0001: new disk, initializing [11322.352936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11322.358082] Lustre: 382857:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11322.432250] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11328.119103] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11330.154706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11330.162618] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11330.236596] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11336.994977] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11340.772458] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11340.789288] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11340.800517] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11340.831174] Lustre: Skipped 1 previous similar message [11347.095852] Lustre: server umount lustre-OST0001 complete [11348.864677] LustreError: 383318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11351.009929] LustreError: 381841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11351.028475] LustreError: 381841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11356.138620] LustreError: 383318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11356.148535] LustreError: 383318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11365.348060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11365.357741] LustreError: Skipped 1 previous similar message [11365.359688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11365.377413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11365.379964] Lustre: Skipped 4 previous similar messages [11366.368731] LustreError: 381843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.386058] LustreError: 381843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11370.666692] Lustre: server umount lustre-OST0000 complete [11376.099382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11376.111380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11376.128569] Lustre: Skipped 1 previous similar message [11380.878260] Lustre: server umount lustre-MDT0000 complete [11384.284426] LustreError: 380002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585365 with bad export cookie 15674877389055922862 [11384.290398] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11384.300989] LustreError: 380002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11390.816677] Lustre: server umount lustre-MDT0001 complete [11401.242290] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [11404.391189] Key type lgssc unregistered [11404.843881] LNet: 384848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11404.852860] LNetError: 384848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11404.871811] LNet: Removed LNI 192.168.206.140@tcp [11405.890231] Key type .llcrypt unregistered [11405.898739] Key type ._llcrypt unregistered [11460.221126] Key type ._llcrypt registered [11460.227572] Key type .llcrypt registered [11460.365403] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [11461.544647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11461.672498] alg: No test for adler32 (adler32-zlib) [11462.675559] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [11462.918194] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [11464.607246] Key type lgssc registered [11465.480973] Lustre: Echo OBD driver; http://www.lustre.org/ [11472.725226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11472.750431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11474.061424] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11474.095452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11474.227653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11478.224201] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11485.958701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11486.093921] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11486.155605] Lustre: 387627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11486.405731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11490.979738] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11497.303962] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11502.033818] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11506.148606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11506.151746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11506.163100] Lustre: Skipped 1 previous similar message [11506.178333] Lustre: Skipped 1 previous similar message [11509.906795] Lustre: server umount lustre-MDT0000 complete [11513.494416] LustreError: 387620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585494 with bad export cookie 9409073101700489722 [11513.496130] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11513.514660] LustreError: 387620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11513.855659] Lustre: server umount lustre-MDT0001 complete [11523.315309] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:31:42 (1787585502) [11529.539963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11530.295721] LustreError: 390139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11530.314278] LustreError: 390139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11530.374786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11534.007116] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11535.847082] LustreError: 390140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11540.962988] LustreError: 390139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11544.960391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11545.985513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11553.096265] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11564.951426] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11572.122393] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11579.848790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11580.182473] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11580.549403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11587.732268] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11598.690891] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11606.472609] Lustre: Permanently deactivating lustre-OST0000 [11606.482541] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11614.748268] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11617.486597] LustreError: 393176:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11617.499118] LustreError: 393176:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11617.506486] LustreError: 393176:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11621.719167] LustreError: 393224:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11621.741458] LustreError: 393224:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11627.214303] Lustre: server umount lustre-OST0000 complete [11630.663534] LustreError: 393474:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11630.672589] LustreError: 393474:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11632.651697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11632.680218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11632.688051] Lustre: Skipped 1 previous similar message [11637.730596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11637.748805] Lustre: Skipped 1 previous similar message [11642.855280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11642.871092] Lustre: Skipped 1 previous similar message [11642.969271] Lustre: server umount lustre-MDT0000 complete [11647.916539] LustreError: 393572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585628 with bad export cookie 9409073101700490303 [11647.928447] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11647.932717] LustreError: 393572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11657.502645] Lustre: server umount lustre-MDT0001 complete [11666.192351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11670.222819] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11673.104539] LustreError: 394636:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11678.517241] Lustre: 394975:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.140@tcp [11680.167848] Lustre: server umount MGS complete [11691.620730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.029930] LustreError: 395698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11692.076344] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11692.082595] Lustre: Skipped 1 previous similar message [11692.125429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11696.475666] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11697.125949] LustreError: 395699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11704.196820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11704.458679] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11709.426454] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11717.457627] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11722.603401] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11724.064732] Lustre: Permanently reactivating lustre-OST0000 [11724.077494] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11724.081693] Lustre: Skipped 2 previous similar messages [11752.928228] LustreError: 397527:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11759.094412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11759.381208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11759.385853] Lustre: Skipped 1 previous similar message [11764.215469] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11772.130501] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11778.021126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11778.021258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11778.036951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11778.055664] Lustre: Skipped 2 previous similar messages [11778.065871] Lustre: Skipped 1 previous similar message [11782.664857] Lustre: server umount lustre-OST0000 complete [11786.722942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11786.738213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.749319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11792.425863] Lustre: server umount lustre-MDT0000 complete [11793.376305] LustreError: 395698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11793.405203] LustreError: 395698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11795.662271] LustreError: 395679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585776 with bad export cookie 9409073101700491353 [11795.666960] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11795.681104] LustreError: 395679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11796.017246] Lustre: server umount lustre-MDT0001 complete [11805.702769] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [11807.978648] Key type lgssc unregistered [11808.256787] LNet: 400161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11808.266630] LNetError: 400161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11808.296383] LNet: Removed LNI 192.168.206.140@tcp [11808.956545] Key type .llcrypt unregistered [11808.958723] Key type ._llcrypt unregistered [11823.853342] Key type ._llcrypt registered [11823.858426] Key type .llcrypt registered [11823.956290] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [11835.501259] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [11836.155612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11836.197048] alg: No test for adler32 (adler32-zlib) [11837.204954] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [11837.477835] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [11839.157046] Key type lgssc registered [11840.177541] Lustre: Echo OBD driver; http://www.lustre.org/ [11871.981693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11871.999414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11873.377972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11873.438117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11873.491856] Lustre: lustre-MDT0000: new disk, initializing [11873.557035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11873.573361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11877.600305] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11890.373312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11890.644264] Lustre: 403298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11890.865441] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11890.873758] Lustre: Skipped 1 previous similar message [11890.974597] Lustre: lustre-MDT0001: new disk, initializing [11891.165864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11891.231507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11891.257223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11898.374479] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11913.325160] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11924.381192] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11933.524520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11933.841739] Lustre: lustre-OST0000: new disk, initializing [11933.845317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11933.855463] Lustre: 405268:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11933.945264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11940.105607] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [11940.545393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11940.566941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11940.640615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11949.490963] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11953.479848] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11953.733552] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11957.726511] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11957.913215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11961.323851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11961.324489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11961.344929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11961.361692] Lustre: Skipped 1 previous similar message [11966.436337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11966.443303] Lustre: Skipped 1 previous similar message [11971.562507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11971.574147] Lustre: Skipped 1 previous similar message [11974.111207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11974.233255] Lustre: server umount lustre-OST0000 complete [11978.211782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11978.218065] LustreError: Skipped 1 previous similar message [11978.228832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11978.244362] Lustre: Skipped 1 previous similar message [11978.250727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11981.814569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11984.201756] Lustre: server umount lustre-MDT0000 complete [11988.403277] LustreError: 403288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585969 with bad export cookie 8260178209436759891 [11988.408461] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11988.414577] LustreError: 403288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11988.722407] Lustre: server umount lustre-MDT0001 complete [11998.357141] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12000.279296] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:39:39 (1787585979) [12007.930952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12008.239635] LustreError: 407730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12008.260871] LustreError: 407730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12008.315153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12012.431666] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12013.544507] LustreError: 407731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12018.659759] LustreError: 407730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12021.851214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12022.203271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12027.886939] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12039.072878] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12046.963093] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12054.111509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12054.427610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12060.616433] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12068.526638] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12070.228962] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12082.345952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12089.324265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12089.328514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12089.341539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12089.362455] Lustre: Skipped 3 previous similar messages [12093.851909] Lustre: server umount lustre-OST0000 complete [12099.042989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12099.049472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12099.056802] Lustre: Skipped 1 previous similar message [12099.060245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12103.618895] Lustre: server umount lustre-MDT0000 complete [12104.672937] LustreError: 407731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12104.706753] LustreError: 407731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12107.458446] LustreError: 407711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586088 with bad export cookie 8260178209436760773 [12107.466445] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12107.476383] LustreError: 407711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12107.851320] Lustre: server umount lustre-MDT0001 complete [12119.651417] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [12122.542685] Key type lgssc unregistered [12122.952037] LNet: 411484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12122.959187] LNetError: 411484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12122.974965] LNet: Removed LNI 192.168.206.140@tcp [12123.844944] Key type .llcrypt unregistered [12123.846596] Key type ._llcrypt unregistered [12148.641975] Key type ._llcrypt registered [12148.645394] Key type .llcrypt registered [12148.750911] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [12149.125829] Key type .llcrypt unregistered [12149.128646] Key type ._llcrypt unregistered [12167.501444] Key type ._llcrypt registered [12167.512135] Key type .llcrypt registered [12167.635773] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [12168.857113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12168.889743] alg: No test for adler32 (adler32-zlib) [12170.186592] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [12170.526288] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [12172.200315] Key type lgssc registered [12173.066499] Lustre: Echo OBD driver; http://www.lustre.org/ [12180.288741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12180.319438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12181.714974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12186.347372] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12191.214456] LustreError: 413700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12191.242225] LustreError: 413700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12195.718599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12195.958219] LustreError: 413701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.979756] LustreError: 413701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12195.988520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12201.447982] LustreError: 413700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.473281] LustreError: 413700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12202.122691] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12206.562460] LustreError: 413701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12206.583255] LustreError: 413701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12206.597272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12209.753228] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12211.690583] LustreError: 413700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12211.714931] LustreError: 413700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12216.701663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12217.045908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12222.571845] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12232.160123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12232.176654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12232.211673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12232.683993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12233.237778] Lustre: server umount lustre-OST0000 complete [12237.802501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12237.802897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12237.813846] Lustre: Skipped 1 previous similar message [12237.819994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12237.832470] Lustre: Skipped 2 previous similar messages [12239.160808] Lustre: server umount lustre-MDT0000 complete [12242.485145] LustreError: 413681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586223 with bad export cookie 13962051760948711014 [12242.494282] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12242.774694] Lustre: server umount lustre-MDT0001 complete [12255.490782] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [12258.905344] Key type lgssc unregistered [12259.190681] LNet: 416848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12259.197518] LNetError: 416848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12259.220679] LNet: Removed LNI 192.168.206.140@tcp [12259.961905] Key type .llcrypt unregistered [12259.968915] Key type ._llcrypt unregistered [12287.600767] Key type ._llcrypt registered [12287.602735] Key type .llcrypt registered [12287.729746] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [12288.785050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12288.804945] alg: No test for adler32 (adler32-zlib) [12289.833668] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [12290.068420] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [12291.791927] Key type lgssc registered [12293.178613] Lustre: Echo OBD driver; http://www.lustre.org/ [12300.861663] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12300.879123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12316.449041] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12321.567291] LustreError: 418240:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [12348.256581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12354.717686] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12363.061706] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12370.101030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12370.736075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12370.819903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12375.397796] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12376.039484] LustreError: 418266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12376.054282] LustreError: 418266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12381.157969] LustreError: 418265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12382.187022] LustreError: 418266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12382.205160] LustreError: 418266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12384.096922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12384.486899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12389.640187] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12397.473212] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12402.836852] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12410.341946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12410.341978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12410.344159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12410.351539] Lustre: Skipped 1 previous similar message [12411.564117] Lustre: server umount lustre-OST0000 complete [12415.465098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12415.468727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12415.477876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12415.483257] Lustre: Skipped 2 previous similar messages [12420.576106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12420.581521] Lustre: Skipped 1 previous similar message [12420.912513] Lustre: server umount lustre-MDT0000 complete [12423.901232] LustreError: 418247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586404 with bad export cookie 12158922276098583849 [12423.905408] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12423.910995] LustreError: 418247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12424.117322] Lustre: server umount lustre-MDT0001 complete [12435.549499] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [12439.320574] Key type lgssc unregistered [12439.637565] LNet: 421813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12439.645205] LNetError: 421813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12439.655542] LNet: Removed LNI 192.168.206.140@tcp [12440.580091] Key type .llcrypt unregistered [12440.585768] Key type ._llcrypt unregistered [12465.475229] Key type ._llcrypt registered [12465.477361] Key type .llcrypt registered [12465.586395] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [12466.443792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12466.481391] alg: No test for adler32 (adler32-zlib) [12467.570682] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [12467.879390] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [12469.664294] Key type lgssc registered [12470.775399] Lustre: Echo OBD driver; http://www.lustre.org/ [12476.882386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12476.896806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12478.222544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12481.873902] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12488.673302] LustreError: 423231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12488.695690] LustreError: 423231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12489.370712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12489.694602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12493.859849] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12501.184993] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12506.428307] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12511.957861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12512.296621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12517.707300] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12523.577019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12523.741153] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12528.610250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12528.612319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12528.615964] Lustre: Skipped 2 previous similar messages [12528.619036] Lustre: Skipped 1 previous similar message [12530.584064] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [12533.602867] Lustre: server umount lustre-MDT0000 complete [12533.729521] LustreError: 424471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12533.760295] LustreError: 424471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12536.213347] LustreError: 423211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.40@tcp arrived at 1787586517 with bad export cookie 2622780177763297149 [12536.251416] LustreError: 423211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12538.601518] LustreError: 423210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586519 with bad export cookie 2622780177763296253 [12538.605666] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12538.612374] LustreError: 423210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12538.851366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12538.852831] LustreError: 423957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12538.884258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12538.902443] LustreError: 423957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12543.968035] LustreError: 423230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12543.970833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12543.987355] LustreError: 423230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12544.913330] Lustre: server umount lustre-OST0000 complete [12549.089679] LustreError: 424471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12549.103535] LustreError: 424471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12552.733169] Lustre: server umount lustre-MDT0001 complete [12563.643644] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [12567.463252] Key type lgssc unregistered [12567.864449] LNet: 426915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12567.877970] LNetError: 426915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12567.901466] LNet: Removed LNI 192.168.206.140@tcp [12568.980186] Key type .llcrypt unregistered [12568.982761] Key type ._llcrypt unregistered [12596.719439] Key type ._llcrypt registered [12596.720748] Key type .llcrypt registered [12596.809489] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [12597.742275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12597.757209] alg: No test for adler32 (adler32-zlib) [12598.790677] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [12599.032185] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [12600.703165] Key type lgssc registered [12601.717525] Lustre: Echo OBD driver; http://www.lustre.org/ [12608.506912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12608.517584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12609.839072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12613.570696] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12619.238218] LustreError: 428333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12619.249196] LustreError: 428333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12621.460561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12621.805531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12626.266390] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12632.100603] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12637.292123] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12643.697465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12643.993780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12649.353696] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12654.264154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12656.014720] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12659.685065] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12659.701913] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12659.708633] Lustre: Skipped 1 previous similar message [12662.686194] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [12664.803636] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12664.812136] Lustre: Skipped 1 previous similar message [12665.312970] Lustre: server umount lustre-MDT0001 complete [12669.927383] LustreError: 428334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12669.959055] LustreError: 428334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12675.040815] LustreError: 428333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12675.049571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12675.056264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12675.090818] Lustre: Skipped 1 previous similar message [12675.096157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12676.659834] Lustre: server umount lustre-OST0000 complete [12680.165627] LustreError: 429072:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12680.815481] Lustre: server umount lustre-MDT0000 complete [12694.820404] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [12697.828938] Key type lgssc unregistered [12698.107481] LNet: 432019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12698.114212] LNetError: 432019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12698.132925] LNet: Removed LNI 192.168.206.140@tcp [12698.993554] Key type .llcrypt unregistered [12698.995663] Key type ._llcrypt unregistered [12722.638444] Key type ._llcrypt registered [12722.643237] Key type .llcrypt registered [12722.755812] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [12737.644253] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [12738.912822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12738.968975] alg: No test for adler32 (adler32-zlib) [12740.083776] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [12740.328983] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [12741.999153] Key type lgssc registered [12743.292283] Lustre: Echo OBD driver; http://www.lustre.org/ [12784.446289] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [12798.054095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12798.082486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12799.356726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12799.382829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12799.464971] Lustre: lustre-MDT0000: new disk, initializing [12799.535900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12799.552115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12803.934208] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12818.993212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12819.165269] Lustre: lustre-OST0000: new disk, initializing [12819.168300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12819.172188] Lustre: Skipped 1 previous similar message [12819.175555] Lustre: 437053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12819.230528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12825.415816] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12828.239701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12828.253199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12828.289353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12834.277437] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12848.936806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12849.068729] Lustre: 436037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12849.126556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12849.180463] Lustre: lustre-MDT0001: new disk, initializing [12849.287411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12849.327169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12849.336138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12853.699913] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [12863.411716] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12863.665251] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12868.243512] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12868.449262] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12874.649433] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [12877.281387] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12877.288799] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12877.292949] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12877.301513] Lustre: Skipped 1 previous similar message [12877.317565] Lustre: Skipped 1 previous similar message [12879.720747] Lustre: server umount lustre-MDT0001 complete [12887.524451] LustreError: 436044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12887.546083] LustreError: 436044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12892.648379] LustreError: 436048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12892.669517] LustreError: 436048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12897.761222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12897.768345] LustreError: 436049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12897.795889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12902.881265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12902.881574] LustreError: 436048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12903.170320] Lustre: server umount lustre-OST0000 complete [12921.311142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12921.828535] Lustre: server umount lustre-MDT0000 complete [12937.051801] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [12940.432883] Key type lgssc unregistered [12940.813217] LNet: 440804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12940.816951] LNetError: 440804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12940.838298] LNet: Removed LNI 192.168.206.140@tcp [12941.832223] Key type .llcrypt unregistered [12941.834810] Key type ._llcrypt unregistered [12981.822262] Key type ._llcrypt registered [12981.825682] Key type .llcrypt registered [12981.919714] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [12982.210830] Key type .llcrypt unregistered [12982.214472] Key type ._llcrypt unregistered [12999.732229] Key type ._llcrypt registered [12999.737037] Key type .llcrypt registered [12999.830093] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [13000.657788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13000.762277] alg: No test for adler32 (adler32-zlib) [13002.081381] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [13002.332143] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [13004.007157] Key type lgssc registered [13005.295489] Lustre: Echo OBD driver; http://www.lustre.org/ [13033.892217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13033.923138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13035.334570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13035.391534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13035.500273] Lustre: lustre-MDT0000: new disk, initializing [13035.618647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13035.640187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13040.850434] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [13052.699933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13052.787806] Lustre: 444183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13052.868595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13052.883050] Lustre: Skipped 1 previous similar message [13052.978921] Lustre: lustre-MDT0001: new disk, initializing [13053.099982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13053.169494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13053.185665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13058.400223] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [13067.556311] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13074.259775] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13080.483457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13080.866174] Lustre: lustre-OST0000: new disk, initializing [13080.869303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13080.874791] Lustre: 446156:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13080.959244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13086.549852] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [13090.380944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13090.407836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13090.592766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13095.981485] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13104.609095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13104.619533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13104.634447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13106.145609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13106.146816] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13106.154934] Lustre: Skipped 1 previous similar message [13108.242466] Lustre: server umount lustre-MDT0000 complete [13111.492137] LustreError: 444175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587092 with bad export cookie 14588625057919349360 [13111.496714] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13111.498027] LustreError: 444175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13111.708474] Lustre: server umount lustre-MDT0001 complete [13125.066990] Lustre: server umount lustre-OST0000 complete [13134.526963] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:58:33 (1787587113) [13142.062384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13142.402421] LustreError: 448181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13142.409275] LustreError: 448181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13142.449204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13146.360799] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [13147.626448] LustreError: 448182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13152.752386] LustreError: 448181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13156.133466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13156.460890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13161.503448] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [13168.910840] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13173.992899] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13193.132243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13193.377749] Lustre: Found index 0 for lustre-OST0000, updating log [13193.537692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13199.523423] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [13207.145252] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13216.067866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13223.394924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13223.413405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13223.441392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13223.460187] Lustre: Skipped 2 previous similar messages [13230.562599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13230.577528] Lustre: Skipped 4 previous similar messages [13236.714662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13236.728084] Lustre: Skipped 1 previous similar message [13237.215117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13237.553766] Lustre: server umount lustre-OST0000 complete [13252.196341] LNet: 1 local NIs in recovery (showing 1): 192.168.206.140@tcp [13253.606720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13253.617987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13253.630279] Lustre: Skipped 1 previous similar message [13253.639328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13258.721065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13259.453816] Lustre: server umount lustre-MDT0000 complete [13262.603389] LustreError: 448162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587243 with bad export cookie 14588625057919350753 [13262.609349] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13262.614523] LustreError: 448162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13262.959371] Lustre: server umount lustre-MDT0001 complete [13272.147962] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:00:50 (1787587250) [13894.542886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13894.984603] LustreError: 454626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13894.998912] LustreError: 454626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13895.044625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13900.127046] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [13900.258199] LustreError: 454627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13900.269487] LustreError: 454627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13904.358524] LustreError: 454626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13908.869597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13909.185529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13913.783281] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [13922.850201] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13929.709819] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13950.897887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13951.257653] Lustre: Found index 0 for lustre-OST0000, updating log [13951.598238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13956.767304] LNet: 1 local NIs in recovery (showing 1): 192.168.206.140@tcp [13958.256298] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [13966.173422] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13968.352730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13968.367080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13968.378044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13968.382307] Lustre: Skipped 2 previous similar messages [13974.435743] Lustre: server umount lustre-OST0000 complete [13979.168426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13979.180035] Lustre: Skipped 1 previous similar message [13979.180276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13979.192960] Lustre: Skipped 1 previous similar message [13983.334379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13983.337416] Lustre: Skipped 6 previous similar messages [13984.473441] Lustre: server umount lustre-MDT0000 complete [13985.384653] LustreError: 454621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13985.410250] LustreError: 454621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13988.558975] LustreError: 457423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587969 with bad export cookie 14588625057919351838 [13988.559413] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13988.570952] LustreError: 457423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13988.937504] Lustre: server umount lustre-MDT0001 complete [14016.159792] LNet: 1 local NIs in recovery (showing 1): 192.168.206.140@tcp [14024.371247] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [14037.684890] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [14078.953566] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:14:18 (1787588058) [14098.919413] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [14101.927437] Key type lgssc unregistered [14102.185667] LNet: 461242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14102.196422] LNetError: 461242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14214.894484] LNet: Removed LNI 192.168.206.140@tcp [14216.127946] Key type .llcrypt unregistered [14216.134158] Key type ._llcrypt unregistered [14232.107591] Key type ._llcrypt registered [14232.109992] Key type .llcrypt registered [14232.192281] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [14233.006055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14233.037166] alg: No test for adler32 (adler32-zlib) [14234.088415] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [14234.311588] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [14235.985307] Key type lgssc registered [14236.916770] Lustre: Echo OBD driver; http://www.lustre.org/ [14241.377673] LNet: Added LNI 192.168.206.140@tcp100 [8/256/0/180] [14242.255749] LNet: Added LNI 192.168.206.140@tcp101 [8/256/0/180] [14242.966566] LNet: Added LNI 192.168.206.140@tcp102 [8/256/0/180] [14243.696798] LNet: Added LNI 192.168.206.140@tcp103 [8/256/0/180] [14244.521722] LNet: Added LNI 192.168.206.140@tcp104 [8/256/0/180] [14245.330296] LNet: Added LNI 192.168.206.140@tcp105 [8/256/0/180] [14246.088897] LNet: Added LNI 192.168.206.140@tcp106 [8/256/0/180] [14246.819899] LNet: Added LNI 192.168.206.140@tcp107 [8/256/0/180] [14247.710058] LNet: Added LNI 192.168.206.140@tcp108 [8/256/0/180] [14248.685942] LNet: Added LNI 192.168.206.140@tcp109 [8/256/0/180] [14249.529595] LNet: Added LNI 192.168.206.140@tcp110 [8/256/0/180] [14250.567467] LNet: Added LNI 192.168.206.140@tcp111 [8/256/0/180] [14251.569736] LNet: Added LNI 192.168.206.140@tcp112 [8/256/0/180] [14252.304278] LNet: Added LNI 192.168.206.140@tcp113 [8/256/0/180] [14253.165722] LNet: Added LNI 192.168.206.140@tcp114 [8/256/0/180] [14253.935843] LNet: Added LNI 192.168.206.140@tcp115 [8/256/0/180] [14254.736079] LNet: Added LNI 192.168.206.140@tcp116 [8/256/0/180] [14255.510268] LNet: Added LNI 192.168.206.140@tcp117 [8/256/0/180] [14256.351177] LNet: Added LNI 192.168.206.140@tcp118 [8/256/0/180] [14257.234135] LNet: Added LNI 192.168.206.140@tcp119 [8/256/0/180] [14258.134096] LNet: Added LNI 192.168.206.140@tcp120 [8/256/0/180] [14258.895520] LNet: Added LNI 192.168.206.140@tcp121 [8/256/0/180] [14259.731979] LNet: Added LNI 192.168.206.140@tcp122 [8/256/0/180] [14260.619433] LNet: Added LNI 192.168.206.140@tcp123 [8/256/0/180] [14261.456573] LNet: Added LNI 192.168.206.140@tcp124 [8/256/0/180] [14262.238560] LNet: Added LNI 192.168.206.140@tcp125 [8/256/0/180] [14263.113312] LNet: Added LNI 192.168.206.140@tcp126 [8/256/0/180] [14264.003936] LNet: Added LNI 192.168.206.140@tcp127 [8/256/0/180] [14264.963779] LNet: Added LNI 192.168.206.140@tcp128 [8/256/0/180] [14265.888437] LNet: Added LNI 192.168.206.140@tcp129 [8/256/0/180] [14266.853658] LNet: Added LNI 192.168.206.140@tcp130 [8/256/0/180] [14267.642730] LNet: Added LNI 192.168.206.140@tcp131 [8/256/0/180] [14268.501490] LNet: Added LNI 192.168.206.140@tcp132 [8/256/0/180] [14269.363916] LNet: Added LNI 192.168.206.140@tcp133 [8/256/0/180] [14270.255173] LNet: Added LNI 192.168.206.140@tcp134 [8/256/0/180] [14271.306344] LNet: Added LNI 192.168.206.140@tcp135 [8/256/0/180] [14278.173374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14278.209820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14279.367796] Lustre: 464276:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.140@tcp: can't add NID 192.168.206.140@tcp130: rc = -75 [14279.452735] LNetError: 464307:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.140 [14279.490275] LNetError: Protocol error connecting to 192.168.206.140@tcp121 at host 192.168.206.140:988: is it running a compatible version of Lustre? [14279.501240] LNetError: Skipped 1 previous similar message [14279.610322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14279.631175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14279.747100] Lustre: lustre-MDT0000: new disk, initializing [14279.841095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14279.857459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14280.415240] LNet: 1 local NIs in recovery (showing 1): 192.168.206.140@tcp121 [14280.416512] LNetError: 464296:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.140 [14280.439824] LNetError: 464296:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14280.458292] LNetError: Protocol error connecting to 192.168.206.140@tcp107 at host 192.168.206.140:988: is it running a compatible version of Lustre? [14280.474092] LNetError: Skipped 2 previous similar messages [14282.477546] LNetError: 464296:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.140 [14282.493658] LNetError: 464296:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14282.511241] LNetError: Protocol error connecting to 192.168.206.140@tcp121 at host 192.168.206.140:988: is it running a compatible version of Lustre? [14282.526483] LNetError: Skipped 4 previous similar messages [14284.558397] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [14286.567070] LNetError: 464310:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.140 [14286.592585] LNetError: Protocol error connecting to 192.168.206.140@tcp121 at host 192.168.206.140:988: is it running a compatible version of Lustre? [14294.754073] LNetError: 464298:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.140 [14294.779226] LNetError: Protocol error connecting to 192.168.206.140@tcp121 at host 192.168.206.140:988: is it running a compatible version of Lustre? [14295.713257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14295.818518] Lustre: 464293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14295.891494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14295.895815] Lustre: Skipped 1 previous similar message [14295.990653] Lustre: lustre-MDT0001: new disk, initializing [14296.070606] Lustre: 465095:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.140@tcp: can't add NID 192.168.206.140@tcp130: rc = -75 [14296.085876] Lustre: 465095:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14296.213618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14296.261638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14296.272448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14300.870647] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [14304.298355] Lustre: 465479:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.140@tcp: can't add NID 192.168.206.140@tcp130: rc = -75 [14304.316267] Lustre: 465479:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14309.091072] LNetError: 464301:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.140 [14309.111751] LNetError: Protocol error connecting to 192.168.206.140@tcp116 at host 192.168.206.140:988: is it running a compatible version of Lustre? [14311.474690] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14319.503721] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14325.474975] LNetError: 464298:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.140 [14325.486489] LNetError: 464298:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14325.515709] LNetError: Protocol error connecting to 192.168.206.140@tcp114 at host 192.168.206.140:988: is it running a compatible version of Lustre? [14325.524294] LNetError: Skipped 10 previous similar messages [14327.875764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14328.219072] Lustre: lustre-OST0000: new disk, initializing [14328.225327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14328.235904] Lustre: 466269:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14328.353242] Lustre: 466274:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.140@tcp: can't add NID 192.168.206.140@tcp130: rc = -75 [14328.360536] Lustre: 466274:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14328.419627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14336.514934] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [14337.139057] Lustre: 466643:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.140@tcp: can't add NID 192.168.206.140@tcp130: rc = -75 [14337.162263] Lustre: 466643:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14337.237262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14337.254901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14337.488461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14340.837161] LNet: 1 local NIs in recovery (showing 1): 192.168.206.140@tcp121 [14348.050742] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14357.988969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14358.001691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14358.019861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14358.242876] LNetError: 464309:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.140 [14358.261481] LNetError: 464309:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 17 previous similar messages [14358.267776] LNetError: Protocol error connecting to 192.168.206.140@tcp126 at host 192.168.206.140:988: is it running a compatible version of Lustre? [14358.276215] LNetError: Skipped 17 previous similar messages [14363.126194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14363.130690] Lustre: Skipped 1 previous similar message [14368.231757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14368.242902] Lustre: Skipped 1 previous similar message [14370.271335] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14370.458042] Lustre: server umount lustre-OST0000 complete [14377.952945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14377.970911] LustreError: Skipped 1 previous similar message [14377.975464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14377.992255] Lustre: Skipped 1 previous similar message [14377.997361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14378.006341] Lustre: Skipped 1 previous similar message [14380.943644] Lustre: server umount lustre-MDT0000 complete [14384.824658] LustreError: 464284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588365 with bad export cookie 1412172799191250333 [14384.830922] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14384.832251] LustreError: 464284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14385.257543] Lustre: server umount lustre-MDT0001 complete [14397.128864] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [14400.329849] Key type lgssc unregistered [14400.714920] LNet: 468188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14400.720860] LNetError: 468188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14401.759159] LNet: Removed LNI 192.168.206.140@tcp [14401.762516] LNetError: 468188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14401.769223] LNet: Removed LNI 192.168.206.140@tcp100 [14402.783237] LNet: Removed LNI 192.168.206.140@tcp101 [14402.799196] LNetError: 468188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14402.807080] LNetError: 468188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14402.817765] LNet: Removed LNI 192.168.206.140@tcp102 [14403.871318] LNet: Removed LNI 192.168.206.140@tcp103 [14404.895172] LNet: Removed LNI 192.168.206.140@tcp104 [14404.905238] LNetError: 468188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14404.913088] LNetError: 468188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14404.917065] LNet: Removed LNI 192.168.206.140@tcp105 [14405.984868] LNet: Removed LNI 192.168.206.140@tcp106 [14407.010076] LNet: Removed LNI 192.168.206.140@tcp107 [14408.031365] LNet: Removed LNI 192.168.206.140@tcp108 [14409.055461] LNet: Removed LNI 192.168.206.140@tcp109 [14409.081575] LNetError: 468188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14409.091761] LNetError: 468188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14409.099734] LNet: Removed LNI 192.168.206.140@tcp110 [14410.143146] LNet: Removed LNI 192.168.206.140@tcp111 [14411.167144] LNet: Removed LNI 192.168.206.140@tcp112 [14412.191157] LNet: Removed LNI 192.168.206.140@tcp113 [14413.215370] LNet: Removed LNI 192.168.206.140@tcp114 [14414.239146] LNet: Removed LNI 192.168.206.140@tcp115 [14415.263250] LNet: Removed LNI 192.168.206.140@tcp116 [14416.287433] LNet: Removed LNI 192.168.206.140@tcp117 [14417.311284] LNet: Removed LNI 192.168.206.140@tcp118 [14417.321224] LNetError: 468188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14417.330679] LNetError: 468188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14418.399297] LNet: Removed LNI 192.168.206.140@tcp119 [14419.423290] LNet: Removed LNI 192.168.206.140@tcp120 [14419.425780] LNet: Removed LNI 192.168.206.140@tcp121 [14420.447785] LNet: Removed LNI 192.168.206.140@tcp122 [14421.471589] LNet: Removed LNI 192.168.206.140@tcp123 [14422.498273] LNet: Removed LNI 192.168.206.140@tcp124 [14423.523591] LNet: Removed LNI 192.168.206.140@tcp125 [14424.543274] LNet: Removed LNI 192.168.206.140@tcp126 [14425.631469] LNet: Removed LNI 192.168.206.140@tcp127 [14426.655435] LNet: Removed LNI 192.168.206.140@tcp128 [14427.679438] LNet: Removed LNI 192.168.206.140@tcp129 [14427.684220] LNet: Removed LNI 192.168.206.140@tcp130 [14427.687639] LNet: Removed LNI 192.168.206.140@tcp131 [14427.690617] LNet: Removed LNI 192.168.206.140@tcp132 [14427.692986] LNet: Removed LNI 192.168.206.140@tcp133 [14427.695135] LNet: Removed LNI 192.168.206.140@tcp134 [14427.703828] LNet: Removed LNI 192.168.206.140@tcp135 [14428.461207] Key type .llcrypt unregistered [14428.463089] Key type ._llcrypt unregistered [14452.216447] Key type ._llcrypt registered [14452.218340] Key type .llcrypt registered [14452.291220] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [14466.875793] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [14468.312464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14468.399319] alg: No test for adler32 (adler32-zlib) [14469.575324] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [14469.875348] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [14471.583310] Key type lgssc registered [14472.745725] Lustre: Echo OBD driver; http://www.lustre.org/ [14512.116175] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:21:31 (1787588491) [14533.109779] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [14536.077693] Key type lgssc unregistered [14536.459626] LNet: 472281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14536.470389] LNetError: 472281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14536.499192] LNet: Removed LNI 192.168.206.140@tcp [14537.589223] Key type .llcrypt unregistered [14537.595808] Key type ._llcrypt unregistered [14555.500569] Key type ._llcrypt registered [14555.505514] Key type .llcrypt registered [14555.673703] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [14557.609422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14557.652782] alg: No test for adler32 (adler32-zlib) [14558.760607] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [14559.031214] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [14560.767334] Key type lgssc registered [14562.187604] Lustre: Echo OBD driver; http://www.lustre.org/ [14571.686244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14571.708396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14573.057958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14573.111428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14573.259743] Lustre: lustre-MDT0000: new disk, initializing [14573.361463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14573.386645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14578.323141] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [14588.674901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14588.849787] Lustre: 473496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14588.935680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14588.939748] Lustre: Skipped 1 previous similar message [14589.009879] Lustre: lustre-MDT0001: new disk, initializing [14589.126622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14589.175886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14589.186266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14594.116258] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [14603.358099] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14608.641407] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14614.553188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14614.790274] Lustre: lustre-OST0000: new disk, initializing [14614.793137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14614.797658] Lustre: 475464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14614.881729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14620.757987] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [14621.259765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14621.275621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14621.408104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14631.133597] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14635.325249] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing lnet_if_list [14640.612917] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14640.649027] LNet: Added LNI 192.168.206.140@tcp10 [8/256/0/180] [14646.222665] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14646.251633] LNet: Added LNI 192.168.206.140@tcp11 [8/256/0/180] [14651.065876] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14651.092988] LNet: Added LNI 192.168.206.140@tcp12 [8/256/0/180] [14655.924150] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14655.956135] LNet: Added LNI 192.168.206.140@tcp13 [8/256/0/180] [14661.158503] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14661.190380] LNet: Added LNI 192.168.206.140@tcp14 [8/256/0/180] [14666.046591] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14666.086305] LNet: Added LNI 192.168.206.140@tcp15 [8/256/0/180] [14671.566558] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14671.610404] LNet: Added LNI 192.168.206.140@tcp16 [8/256/0/180] [14677.369804] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14677.402666] LNet: Added LNI 192.168.206.140@tcp17 [8/256/0/180] [14681.865236] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14681.892527] LNet: Added LNI 192.168.206.140@tcp18 [8/256/0/180] [14687.735194] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14687.769543] LNet: Added LNI 192.168.206.140@tcp19 [8/256/0/180] [14693.627318] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14693.671859] LNet: Added LNI 192.168.206.140@tcp20 [8/256/0/180] [14700.358242] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14700.402657] LNet: Added LNI 192.168.206.140@tcp21 [8/256/0/180] [14707.214711] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14707.243386] LNet: Added LNI 192.168.206.140@tcp22 [8/256/0/180] [14712.991987] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14713.023143] LNet: Added LNI 192.168.206.140@tcp23 [8/256/0/180] [14720.420504] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14720.448740] LNet: Added LNI 192.168.206.140@tcp24 [8/256/0/180] [14727.529124] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14727.569919] LNet: Added LNI 192.168.206.140@tcp25 [8/256/0/180] [14734.063256] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14734.094453] LNet: Added LNI 192.168.206.140@tcp26 [8/256/0/180] [14739.889678] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14739.915932] LNet: Added LNI 192.168.206.140@tcp27 [8/256/0/180] [14745.967870] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14746.027336] LNet: Added LNI 192.168.206.140@tcp28 [8/256/0/180] [14752.614475] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14752.654397] LNet: Added LNI 192.168.206.140@tcp29 [8/256/0/180] [14758.927502] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14758.952285] LNet: Added LNI 192.168.206.140@tcp30 [8/256/0/180] [14765.384609] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14765.410939] LNet: Added LNI 192.168.206.140@tcp31 [8/256/0/180] [14771.885337] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14771.914398] LNet: Added LNI 192.168.206.140@tcp32 [8/256/0/180] [14778.708276] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14778.758286] LNet: Added LNI 192.168.206.140@tcp33 [8/256/0/180] [14785.625179] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14785.658991] LNet: Added LNI 192.168.206.140@tcp34 [8/256/0/180] [14792.808969] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14792.858933] LNet: Added LNI 192.168.206.140@tcp35 [8/256/0/180] [14800.708144] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14800.740276] LNet: Added LNI 192.168.206.140@tcp36 [8/256/0/180] [14808.155905] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14808.199795] LNet: Added LNI 192.168.206.140@tcp37 [8/256/0/180] [14816.677693] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14816.722151] LNet: Added LNI 192.168.206.140@tcp38 [8/256/0/180] [14825.467329] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14825.519046] LNet: Added LNI 192.168.206.140@tcp39 [8/256/0/180] [14835.269659] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14835.301359] LNet: Added LNI 192.168.206.140@tcp40 [8/256/0/180] [14843.422229] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14843.460454] LNet: Added LNI 192.168.206.140@tcp41 [8/256/0/180] [14852.329397] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14852.363747] LNet: Added LNI 192.168.206.140@tcp42 [8/256/0/180] [14860.548374] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14860.599624] LNet: Added LNI 192.168.206.140@tcp43 [8/256/0/180] [14868.808335] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14868.841637] LNet: Added LNI 192.168.206.140@tcp44 [8/256/0/180] [14876.403334] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14876.444914] LNet: Added LNI 192.168.206.140@tcp45 [8/256/0/180] [14883.538818] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14883.581820] LNet: Added LNI 192.168.206.140@tcp46 [8/256/0/180] [14890.502471] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14890.524951] LNet: Added LNI 192.168.206.140@tcp47 [8/256/0/180] [14897.232093] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14897.264337] LNet: Added LNI 192.168.206.140@tcp48 [8/256/0/180] [14906.521811] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14906.567199] LNet: Added LNI 192.168.206.140@tcp49 [8/256/0/180] [14915.865306] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14915.913045] LNet: Added LNI 192.168.206.140@tcp50 [8/256/0/180] [14937.568845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14937.584853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14937.602246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14939.118632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14939.120278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14939.150249] Lustre: Skipped 2 previous similar messages [14941.312824] Lustre: server umount lustre-MDT0000 complete [14945.031402] LustreError: 473487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588925 with bad export cookie 8337158485641816766 [14945.043219] LustreError: 473487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14945.043990] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14945.364834] Lustre: server umount lustre-MDT0001 complete [14963.736866] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [14965.731202] Lustre: 472767:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787588930/real 1787588930] req@00000000e9cf7c74 x1874422440045568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787588946 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14965.771482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14965.782183] Lustre: Skipped 1 previous similar message [14966.757868] Lustre: 472768:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787588931/real 1787588931] req@000000000ecac0df x1874422440045696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787588947 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14970.859980] Lustre: 472769:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787588935/real 1787588935] req@00000000bc3ad399 x1874422440045824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787588951 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14972.985051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14975.968293] Lustre: 472767:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787588940/real 1787588940] req@00000000bd0ebbac x1874422440046080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787588956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14980.130152] Lustre: 472767:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787588945/real 1787588945] req@00000000a2c1bf2b x1874422440046208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787588961 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14989.564687] LustreError: 475456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14989.582485] LustreError: 475456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14989.632872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14990.635385] LustreError: 499706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14990.666663] LustreError: 499706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14994.644183] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [14994.743524] LustreError: 499708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14994.750646] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14994.760018] LustreError: 499708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15000.042047] LustreError: 499706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15003.910895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15004.320534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15009.590471] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15009.782701] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15017.610095] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15023.707350] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15045.089547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15045.093864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15045.113021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15048.828305] Lustre: server umount lustre-OST0000 complete [15055.328425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15055.333599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15055.351269] Lustre: Skipped 1 previous similar message [15055.364285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15055.367487] Lustre: Skipped 1 previous similar message [15059.071855] Lustre: server umount lustre-MDT0000 complete [15060.960470] LustreError: 499712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15060.976915] LustreError: 499712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15063.070398] LustreError: 474290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589043 with bad export cookie 8337158485641819342 [15063.075868] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15063.080335] LustreError: 474290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15063.402969] Lustre: server umount lustre-MDT0001 complete [15075.669533] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [15078.620218] Key type lgssc unregistered [15079.011657] LNet: 502493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15079.026403] LNetError: 502493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15079.038801] LNet: Removed LNI 192.168.206.140@tcp [15079.047393] LNet: Removed LNI 192.168.206.140@tcp10 [15079.058236] LNet: Removed LNI 192.168.206.140@tcp11 [15079.071265] LNet: Removed LNI 192.168.206.140@tcp12 [15079.083364] LNet: Removed LNI 192.168.206.140@tcp13 [15079.087781] LNet: Removed LNI 192.168.206.140@tcp14 [15079.090023] LNet: Removed LNI 192.168.206.140@tcp15 [15079.091776] LNet: Removed LNI 192.168.206.140@tcp16 [15079.094965] LNet: Removed LNI 192.168.206.140@tcp17 [15079.096863] LNet: Removed LNI 192.168.206.140@tcp18 [15079.105458] LNet: Removed LNI 192.168.206.140@tcp19 [15079.109339] LNet: Removed LNI 192.168.206.140@tcp20 [15079.113367] LNet: Removed LNI 192.168.206.140@tcp21 [15079.117305] LNet: Removed LNI 192.168.206.140@tcp22 [15079.121686] LNet: Removed LNI 192.168.206.140@tcp23 [15079.128300] LNet: Removed LNI 192.168.206.140@tcp24 [15079.135459] LNet: Removed LNI 192.168.206.140@tcp25 [15079.141623] LNet: Removed LNI 192.168.206.140@tcp26 [15079.145720] LNet: Removed LNI 192.168.206.140@tcp27 [15079.149830] LNet: Removed LNI 192.168.206.140@tcp28 [15079.153514] LNet: Removed LNI 192.168.206.140@tcp29 [15079.156878] LNet: Removed LNI 192.168.206.140@tcp30 [15079.159762] LNet: Removed LNI 192.168.206.140@tcp31 [15079.162383] LNet: Removed LNI 192.168.206.140@tcp32 [15079.164959] LNet: Removed LNI 192.168.206.140@tcp33 [15079.166855] LNet: Removed LNI 192.168.206.140@tcp34 [15079.172309] LNet: Removed LNI 192.168.206.140@tcp35 [15079.187413] LNet: Removed LNI 192.168.206.140@tcp36 [15079.192973] LNet: Removed LNI 192.168.206.140@tcp37 [15079.196289] LNet: Removed LNI 192.168.206.140@tcp38 [15079.202097] LNet: Removed LNI 192.168.206.140@tcp39 [15079.212148] LNet: Removed LNI 192.168.206.140@tcp40 [15079.217063] LNet: Removed LNI 192.168.206.140@tcp41 [15079.221610] LNet: Removed LNI 192.168.206.140@tcp42 [15079.224287] LNet: Removed LNI 192.168.206.140@tcp43 [15079.227338] LNet: Removed LNI 192.168.206.140@tcp44 [15079.229866] LNet: Removed LNI 192.168.206.140@tcp45 [15079.233075] LNet: Removed LNI 192.168.206.140@tcp46 [15079.237580] LNet: Removed LNI 192.168.206.140@tcp47 [15079.239461] LNet: Removed LNI 192.168.206.140@tcp48 [15079.244322] LNet: Removed LNI 192.168.206.140@tcp49 [15079.269047] LNet: Removed LNI 192.168.206.140@tcp50 [15080.134181] Key type .llcrypt unregistered [15080.138748] Key type ._llcrypt unregistered [15100.811137] Key type ._llcrypt registered [15100.815728] Key type .llcrypt registered [15100.907302] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [15115.526190] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [15116.609151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15116.630838] alg: No test for adler32 (adler32-zlib) [15117.851966] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [15118.166166] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [15119.831157] Key type lgssc registered [15120.800989] Lustre: Echo OBD driver; http://www.lustre.org/ [15156.966947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15157.015635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15158.410129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15158.471893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15158.542544] Lustre: lustre-MDT0000: new disk, initializing [15158.601249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15158.621477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15163.642703] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15175.244355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15175.366749] Lustre: 505693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15175.417603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15175.420828] Lustre: Skipped 1 previous similar message [15175.484095] Lustre: lustre-MDT0001: new disk, initializing [15175.575316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15175.613225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15175.622698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15180.632632] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15190.723350] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15197.313663] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15205.216504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15205.399677] Lustre: *** cfs_fail_loc=90f, val=0*** [15205.503341] Lustre: lustre-OST0000: new disk, initializing [15205.510974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15205.525182] Lustre: 507709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15205.656368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15212.502850] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15212.638266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15212.646246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15212.801537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15222.388421] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15233.506194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15233.525665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15233.525780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15236.548296] Lustre: server umount lustre-OST0000 complete [15242.211447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15242.225417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15242.261825] Lustre: Skipped 1 previous similar message [15242.275597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15242.288673] Lustre: Skipped 1 previous similar message [15243.760930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15243.763035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15246.880726] Lustre: server umount lustre-MDT0000 complete [15250.996450] LustreError: 505684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589231 with bad export cookie 16805089632139029047 [15251.009935] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15251.016845] LustreError: 505684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15251.367216] Lustre: server umount lustre-MDT0001 complete [15264.239358] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [15267.666993] Key type lgssc unregistered [15267.997070] LNet: 509617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15268.025785] LNetError: 509617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15268.047309] LNet: Removed LNI 192.168.206.140@tcp [15269.150824] Key type .llcrypt unregistered [15269.154890] Key type ._llcrypt unregistered [15288.453456] Key type ._llcrypt registered [15288.457676] Key type .llcrypt registered [15288.606823] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [15289.700187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15289.872105] alg: No test for adler32 (adler32-zlib) [15290.972172] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [15291.246678] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [15292.927207] Key type lgssc registered [15294.205208] Lustre: Echo OBD driver; http://www.lustre.org/ [15303.193077] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15303.256320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15304.669782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15308.902708] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15314.914503] LustreError: 510886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15314.962034] LustreError: 510886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15318.148886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15318.514968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15323.240370] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15331.406753] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15337.987308] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15345.124309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15345.526931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15351.957481] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15361.258356] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15375.087093] loop0: detected capacity change from 0 to 204800000 [15375.305805] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15375.322532] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15375.334260] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15382.190894] loop0: detected capacity change from 0 to 204800000 [15382.460793] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15382.478164] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15382.488916] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15388.858271] loop0: detected capacity change from 0 to 204800000 [15395.351611] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15395.591207] Lustre: fs15246-OST0000: new disk, initializing [15395.599816] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15395.610767] Lustre: 514483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15395.727423] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15403.259335] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15413.513702] loop1: detected capacity change from 0 to 204800000 [15419.730991] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15419.888262] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15419.984055] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15420.099536] Lustre: fs15246-MDT0000: new disk, initializing [15420.321215] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15420.377067] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15420.533628] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15421.602109] Lustre: 515606:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000079b8270a x1874423208525568/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787589418 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15425.521263] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15426.672549] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15426.682973] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15426.749115] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15452.143969] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15452.160242] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15453.936665] Lustre: server umount fs15246-MDT0000 complete [15457.275619] LustreError: 510883:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15462.377431] LustreError: 510883:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15465.110653] Lustre: server umount fs15246-OST0000 complete [15481.828735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15481.846654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15481.878271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15482.864602] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15482.870285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15482.888166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15482.918285] Lustre: Skipped 1 previous similar message [15484.672520] Lustre: server umount lustre-OST0000 complete [15492.577215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15492.587328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15492.597399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15495.574204] Lustre: server umount lustre-MDT0000 complete [15498.213694] LustreError: 510886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15498.229983] LustreError: 510886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15499.703475] LustreError: 510866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589480 with bad export cookie 6363094821334967445 [15499.721557] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15499.722862] LustreError: 510866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15500.123930] Lustre: server umount lustre-MDT0001 complete [15511.630677] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [15514.335726] Key type lgssc unregistered [15514.628671] LNet: 518709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15514.638658] LNetError: 518709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15514.655887] LNet: Removed LNI 192.168.206.140@tcp [15515.650434] Key type .llcrypt unregistered [15515.652059] Key type ._llcrypt unregistered [15533.425337] Key type ._llcrypt registered [15533.427451] Key type .llcrypt registered [15533.507698] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [15534.497770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15534.645261] alg: No test for adler32 (adler32-zlib) [15535.710225] Lustre: Lustre: Build Version: 2.17.57_80_g9b8a413 [15535.935385] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [15537.623157] Key type lgssc registered [15538.827660] Lustre: Echo OBD driver; http://www.lustre.org/ [15583.351238] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [15598.077651] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [15634.660276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15634.710335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15636.049147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15636.122916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15636.179388] Lustre: lustre-MDT0000: new disk, initializing [15636.346843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15636.364757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15640.906261] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15652.904151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15653.129073] Lustre: 523199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15653.255600] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15653.261752] Lustre: Skipped 1 previous similar message [15653.397234] Lustre: lustre-MDT0001: new disk, initializing [15653.537287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15653.606722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15653.621865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15659.825551] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15671.224390] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15677.961405] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15685.586065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15685.896984] Lustre: lustre-OST0000: new disk, initializing [15685.900066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15685.909849] Lustre: 525167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15686.000213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15692.857681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15692.875231] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15692.969555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15693.228091] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [15702.832837] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15708.493551] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15708.746778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15713.866439] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15714.106544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15718.886763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15718.895773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15718.906874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15718.912028] Lustre: Skipped 1 previous similar message [15718.926724] LustreError: Skipped 1 previous similar message [15722.249610] Lustre: server umount lustre-OST0000 complete [15729.122551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15729.129174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15729.147482] Lustre: Skipped 1 previous similar message [15729.156638] Lustre: Skipped 2 previous similar messages [15734.240394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15734.256966] Lustre: Skipped 1 previous similar message [15738.339899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15738.346869] Lustre: Skipped 1 previous similar message [15740.383487] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15740.620924] Lustre: server umount lustre-MDT0000 complete [15744.241073] LustreError: 523191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589725 with bad export cookie 7758736243886730087 [15744.247438] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15744.252391] LustreError: 523191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15744.529825] Lustre: server umount lustre-MDT0001 complete [15771.610713] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [15782.855221] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [15817.838167] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15541 sec ========= 12:43:16 (1787589796) [15819.804222] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:43:18 (1787589798) === [15823.291841] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:43:22 (1787589802) === [15869.065603] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [15872.207232] Key type lgssc unregistered [15872.604977] LNet: 531818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15872.612603] LNetError: 531818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15872.628793] LNet: Removed LNI 192.168.206.140@tcp [15873.642640] Key type .llcrypt unregistered [15873.644284] Key type ._llcrypt unregistered