[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 465161434 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002389] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006829] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.010117] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.014056] *** VALIDATE selinux *** [ 0.022059] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027653] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031107] *** VALIDATE tmpfs *** [ 0.033465] *** VALIDATE proc *** [ 0.035076] *** VALIDATE cgroup *** [ 0.036017] *** VALIDATE cgroup2 *** [ 0.037236] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038117] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040035] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.043821] debug: unmapping init [mem 0xffffffffb3c59000-0xffffffffb3c60fff] [ 0.045229] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046614] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049026] ... generic registers: 4 [ 0.050009] ... value mask: 0000ffffffffffff [ 0.051010] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054320] rcu: Hierarchical SRCU implementation. [ 0.056419] smp: Bringing up secondary CPUs ... [ 0.057624] x86: Booting SMP configuration: [ 0.058029] .... node #0, CPUs: #1 #2 #3 [ 0.063033] smp: Brought up 1 node, 4 CPUs [ 0.065013] smpboot: Max logical packages: 1 [ 0.066013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132822] node 0 deferred pages initialised in 61ms [ 0.135237] devtmpfs: initialized [ 0.136229] x86/mm: Memory block size: 128MB [ 0.138209] gcov: version magic: 0x41383552 [ 0.139568] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146290] pinctrl core: initialized pinctrl subsystem [ 0.148157] [ 0.148754] ************************************************************* [ 0.151013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153009] ** ** [ 0.155011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157010] ** ** [ 0.159013] ** This means that this kernel is built to expose internal ** [ 0.162014] ** IOMMU data structures, which may compromise security on ** [ 0.164012] ** your system. ** [ 0.166011] ** ** [ 0.168011] ** If you see this message and you are not debugging the ** [ 0.170010] ** kernel, report this immediately to your vendor! ** [ 0.172010] ** ** [ 0.174009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175010] ************************************************************* [ 0.178622] NET: Registered protocol family 16 [ 0.180512] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.186077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.190018] cpuidle: using governor menu [ 0.191623] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194783] PCI: Using configuration type 1 for base access [ 0.196123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211073] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.216045] cryptd: max_cpu_qlen set to 1000 [ 0.219689] ACPI: Added _OSI(Module Device) [ 0.221016] ACPI: Added _OSI(Processor Device) [ 0.223014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230196] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237008] ACPI: Interpreter enabled [ 0.238057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240027] ACPI: Using IOAPIC for interrupt routing [ 0.241000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243297] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254370] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268447] acpiphp: Slot [2] registered [ 0.270110] acpiphp: Slot [5] registered [ 0.271119] acpiphp: Slot [6] registered [ 0.273118] acpiphp: Slot [7] registered [ 0.274102] acpiphp: Slot [8] registered [ 0.276100] acpiphp: Slot [9] registered [ 0.278154] acpiphp: Slot [10] registered [ 0.279149] acpiphp: Slot [3] registered [ 0.281085] acpiphp: Slot [4] registered [ 0.282081] acpiphp: Slot [11] registered [ 0.284080] acpiphp: Slot [12] registered [ 0.285090] acpiphp: Slot [13] registered [ 0.287078] acpiphp: Slot [14] registered [ 0.288096] acpiphp: Slot [15] registered [ 0.289098] acpiphp: Slot [16] registered [ 0.291119] acpiphp: Slot [17] registered [ 0.294124] acpiphp: Slot [18] registered [ 0.296103] acpiphp: Slot [19] registered [ 0.297097] acpiphp: Slot [20] registered [ 0.299095] acpiphp: Slot [21] registered [ 0.300157] acpiphp: Slot [22] registered [ 0.302121] acpiphp: Slot [23] registered [ 0.304133] acpiphp: Slot [24] registered [ 0.305131] acpiphp: Slot [25] registered [ 0.306099] acpiphp: Slot [26] registered [ 0.308102] acpiphp: Slot [27] registered [ 0.309124] acpiphp: Slot [28] registered [ 0.311103] acpiphp: Slot [29] registered [ 0.312090] acpiphp: Slot [30] registered [ 0.314121] acpiphp: Slot [31] registered [ 0.315059] PCI host bridge to bus 0000:00 [ 0.316014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.319019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.321019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.323022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.325020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.328035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.330180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.334564] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.338507] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.349014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353620] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.355018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.357018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.360030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362545] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.365943] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.368030] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372091] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.392016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.397011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.403091] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.419015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.439015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.451734] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.461015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.496000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.509043] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.520014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.527014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.546017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.556112] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.565025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.572013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.595017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.608515] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.617013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.623016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.640016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.657932] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.668015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.679027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.712016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.728000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.731378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.734366] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.737335] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.739259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.746055] iommu: Default domain type: Passthrough [ 0.748399] SCSI subsystem initialized [ 0.749121] ACPI: bus type USB registered [ 0.751095] usbcore: registered new interface driver usbfs [ 0.753070] usbcore: registered new interface driver hub [ 0.755102] usbcore: registered new device driver usb [ 0.757199] pps_core: LinuxPPS API ver. 1 registered [ 0.759010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.761057] PTP clock support registered [ 0.764067] EDAC MC: Ver: 3.0.0 [ 0.765501] PCI: Using ACPI for IRQ routing [ 0.766981] NetLabel: Initializing [ 0.767010] NetLabel: domain hash size = 128 [ 0.768007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.769126] NetLabel: unlabeled traffic allowed by default [ 0.771274] vgaarb: loaded [ 0.773442] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.777015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.783186] clocksource: Switched to clocksource kvm-clock [ 0.898223] VFS: Disk quotas dquot_6.6.0 [ 0.900052] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.902742] *** VALIDATE ramfs *** [ 0.904177] *** VALIDATE hugetlbfs *** [ 0.905891] pnp: PnP ACPI init [ 0.908608] pnp: PnP ACPI: found 6 devices [ 0.937053] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.941770] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.944279] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.946839] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.949566] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.952593] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.955395] NET: Registered protocol family 2 [ 0.958101] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.963390] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.967563] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.973049] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.977457] TCP: Hash tables configured (established 65536 bind 65536) [ 0.981582] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.985757] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.989438] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.992729] NET: Registered protocol family 1 [ 0.996307] RPC: Registered named UNIX socket transport module. [ 0.997706] RPC: Registered udp transport module. [ 0.999657] RPC: Registered tcp transport module. [ 1.001530] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.003778] NET: Registered protocol family 44 [ 1.005050] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.007072] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.008820] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.010911] PCI: CLS 0 bytes, default 64 [ 1.012146] Unpacking initramfs... [ 2.403924] debug: unmapping init [mem 0xffff8e01fcc54000-0xffff8e01fffbffff] [ 2.408534] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.410713] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.413587] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.925960] Initialise system trusted keyrings [ 2.928243] Key type blacklist registered [ 2.931125] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.941368] zbud: loaded [ 2.944702] *** VALIDATE nfs *** [ 2.946556] *** VALIDATE nfs4 *** [ 2.948362] pstore: using deflate compression [ 2.952385] Platform Keyring initialized [ 3.089067] NET: Registered protocol family 38 [ 3.091643] Key type asymmetric registered [ 3.093569] Asymmetric key parser 'x509' registered [ 3.096875] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.099732] io scheduler mq-deadline registered [ 3.101136] io scheduler kyber registered [ 3.102434] io scheduler bfq registered [ 3.104366] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.107910] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.110408] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.112825] ACPI: Power Button [PWRF] [ 3.117810] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.124684] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.142803] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.152415] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.175683] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.207502] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.235695] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.241381] Non-volatile memory driver v1.3 [ 3.242642] Linux agpgart interface v0.103 [ 3.269665] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.272861] vda: detected capacity change from 0 to 76779520 [ 3.288664] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.292977] vdb: detected capacity change from 0 to 1073741824 [ 3.313619] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.317030] vdc: detected capacity change from 0 to 2621440000 [ 3.336501] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.340923] vdd: detected capacity change from 0 to 2621440000 [ 3.358966] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.361982] vde: detected capacity change from 0 to 4294967296 [ 3.379544] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.383032] vdf: detected capacity change from 0 to 4294967296 [ 3.392436] libphy: Fixed MDIO Bus: probed [ 3.397549] usbcore: registered new interface driver usbserial_generic [ 3.399988] usbserial: USB Serial support registered for generic [ 3.402483] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.406849] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.410858] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.413496] mousedev: PS/2 mouse device common for all mice [ 3.417234] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.418061] rtc_cmos 00:05: RTC can wake from S4 [ 3.427767] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.429317] rtc_cmos 00:05: registered as rtc0 [ 3.437966] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.440444] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.443402] intel_pstate: CPU model not supported [ 3.453543] hid: raw HID events driver (C) Jiri Kosina [ 3.461334] usbcore: registered new interface driver usbhid [ 3.463696] usbhid: USB HID core driver [ 3.466704] drop_monitor: Initializing network drop monitor service [ 3.469427] Initializing XFRM netlink socket [ 3.471647] NET: Registered protocol family 10 [ 3.475363] Segment Routing with IPv6 [ 3.477788] NET: Registered protocol family 17 [ 3.482483] mpls_gso: MPLS GSO support [ 3.493612] RAS: Correctable Errors collector initialized. [ 3.495902] AVX version of gcm_enc/dec engaged. [ 3.498378] AES CTR mode by8 optimization enabled [ 3.596746] sched_clock: Marking stable (3596722928, 0)->(4433283566, -836560638) [ 3.600720] registered taskstats version 1 [ 3.603481] Loading compiled-in X.509 certificates [ 3.605842] zswap: loaded using pool lzo/zbud [ 3.631808] Key type big_key registered [ 3.643928] Key type encrypted registered [ 3.645414] ima: No TPM chip found, activating TPM-bypass! [ 3.647085] ima: Allocated hash algorithm: sha1 [ 3.648560] ima: No architecture policies found [ 3.649996] evm: Initialising EVM extended attributes: [ 3.651576] evm: security.selinux [ 3.652576] evm: security.ima [ 3.653489] evm: security.capability [ 3.654517] evm: HMAC attrs: 0x1 [ 3.657338] rtc_cmos 00:05: setting system clock to 2026-09-15 02:03:56 UTC (1789437836) [ 3.665275] debug: unmapping init [mem 0xffffffffb4c03000-0xffffffffb4dfffff] [ 3.668785] debug: unmapping init [mem 0xffffffffb3982000-0xffffffffb3c58fff] [ 3.678137] Write protecting the kernel read-only data: 28672k [ 3.681618] debug: unmapping init [mem 0xffffffffb2003000-0xffffffffb21fffff] [ 3.684028] debug: unmapping init [mem 0xffffffffb2914000-0xffffffffb29fffff] [ 3.732301] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.740661] systemd[1]: Detected virtualization kvm. [ 3.743254] systemd[1]: Detected architecture x86-64. [ 3.745100] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.774947] systemd[1]: No hostname configured. [ 3.777147] systemd[1]: Set hostname to . [ 3.779485] random: systemd: uninitialized urandom read (16 bytes read) [ 3.782257] systemd[1]: Initializing machine ID from random generator. [ 3.952140] random: systemd: uninitialized urandom read (16 bytes read) [ 3.955208] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.960585] random: systemd: uninitialized urandom read (16 bytes read) [ 3.963728] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.968687] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.690934] device-mapper: uevent: version 1.0.3 [ 4.693451] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.489989] virtio_net virtio0 ens2: renamed from eth0 [ 5.527735] random: fast init done [ 5.588596] scsi host0: ata_piix [ 5.665257] scsi host1: ata_piix [ 5.698745] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.703681] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.399666] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.160156] random: crng init done [ 10.161797] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.799799] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.096770] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.378098] SELinux: Disabled at runtime. [ 12.440335] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.451695] systemd[1]: Detected virtualization kvm. [ 12.454170] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.045911] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.050242] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.056820] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.061369] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.065156] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.075461] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.080729] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 13.249800] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.657029] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.952510] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.024576] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.208799] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.222924] EDAC sbridge: Ver: 1.1.2 [ 16.216647] Key type dns_resolver registered [ 16.575396] NFS: Registering the id_resolver key type [ 16.577642] Key type id_resolver registered [ 16.580663] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg615-server login: [ 35.599544] hrtimer: interrupt took 7554190 ns [ 61.364510] libcfs: loading out-of-tree module taints kernel. [ 61.478956] Key type ._llcrypt registered [ 61.480314] Key type .llcrypt registered [ 61.562318] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 83.891910] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 85.628561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 85.652235] alg: No test for adler32 (adler32-zlib) [ 87.247338] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 88.232334] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 89.952160] Key type lgssc registered [ 92.030152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 109.268722] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 158.065547] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 172.817617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 172.872639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.174728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 174.239672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 174.357576] Lustre: lustre-MDT0000: new disk, initializing [ 174.507890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.533290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 179.262534] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.182708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.309500] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.345318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 194.351332] Lustre: Skipped 1 previous similar message [ 194.444156] Lustre: lustre-MDT0001: new disk, initializing [ 194.517197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 194.551823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 194.563513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 199.233319] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.831802] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.555396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.863262] Lustre: lustre-OST0000: new disk, initializing [ 215.866207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.872709] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.935962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.685801] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.743476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 222.752103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 222.883180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 238.970855] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.191900] Lustre: lustre-OST0001: new disk, initializing [ 239.197201] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.220600] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.350818] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.304916] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.901788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 248.918836] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 248.987105] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.674955] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 268.965324] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.264879] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing check_logdir /tmp/testlogs/ [ 281.128670] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing yml_node [ 285.937946] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 288.198798] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 290.827653] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 292.411674] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 14 22:08:44 EDT 2026 [ 312.570397] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 314.586249] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.722221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.723585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.743346] Lustre: Skipped 3 previous similar messages [ 330.755549] Lustre: Skipped 3 previous similar messages [ 335.848861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.858750] Lustre: Skipped 3 previous similar messages [ 336.455291] Lustre: server umount lustre-MDT0000 complete [ 340.168094] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438173 with bad export cookie 1391483320187758488 [ 340.170973] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.184980] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 340.961888] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.992417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.997555] Lustre: Skipped 1 previous similar message [ 346.101330] LustreError: 6519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.101497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.140722] LustreError: 6519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 346.158400] Lustre: Skipped 1 previous similar message [ 346.690704] Lustre: server umount lustre-MDT0001 complete [ 351.033251] Lustre: server umount lustre-OST0000 complete [ 355.168476] Lustre: server umount lustre-OST0001 complete [ 361.587205] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 371.582745] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 408.260583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.456663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 408.496211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 408.560373] Lustre: lustre-MDT0000: new disk, initializing [ 408.663567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 408.691619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 413.513323] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 424.375353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.514505] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 424.653353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 424.662598] Lustre: Skipped 1 previous similar message [ 424.766982] Lustre: lustre-MDT0001: new disk, initializing [ 424.937458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 424.981559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 424.992587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 430.209953] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 440.262359] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 446.262350] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 453.275932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 453.640417] Lustre: lustre-OST0000: new disk, initializing [ 453.644209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 453.651567] Lustre: 17044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 453.749240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 455.636406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 455.645697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 455.805317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 460.212861] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 469.808650] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 475.196725] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 475.473525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 480.025565] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 480.281539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 486.373630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.376384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 486.394493] Lustre: Skipped 1 previous similar message [ 486.396363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 486.440039] LustreError: Skipped 1 previous similar message [ 496.616340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.637951] Lustre: Skipped 4 previous similar messages [ 497.120123] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 497.422448] Lustre: server umount lustre-OST0000 complete [ 506.851911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.852785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 506.868201] Lustre: Skipped 1 previous similar message [ 509.877161] Lustre: server umount lustre-MDT0000 complete [ 511.983854] LustreError: 17631:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.028894] LustreError: 17631:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 514.778074] LustreError: 17043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438347 with bad export cookie 1391483320187760021 [ 514.782821] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 514.804835] LustreError: 17043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 515.285682] Lustre: server umount lustre-MDT0001 complete [ 520.666945] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:12:32 (1789438352) [ 523.552974] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 525.428512] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:12:37 (1789438357) [ 533.476798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.051342] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.105804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.422602] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 539.105420] LustreError: 19457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.210065] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.211233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 552.196079] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 560.832674] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 567.298976] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 574.846263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 583.096475] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 592.413617] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 603.105501] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.117085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.150555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 603.159176] Lustre: Skipped 2 previous similar messages [ 603.177908] Lustre: Skipped 3 previous similar messages [ 606.730823] Lustre: server umount lustre-OST0000 complete [ 613.345863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.381727] Lustre: Skipped 1 previous similar message [ 618.003748] Lustre: server umount lustre-MDT0000 complete [ 618.470027] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.488535] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 623.784454] LustreError: 22037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438456 with bad export cookie 1391483320187760903 [ 623.785688] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.804958] LustreError: 22037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 624.715672] Lustre: server umount lustre-MDT0001 complete [ 639.773910] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 643.166746] Key type lgssc unregistered [ 643.551725] LNet: 22990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.557391] LNetError: 22990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.577717] LNet: Removed LNI 192.168.206.115@tcp [ 644.547478] Key type .llcrypt unregistered [ 644.554420] Key type ._llcrypt unregistered [ 668.404365] Key type ._llcrypt registered [ 668.407755] Key type .llcrypt registered [ 668.511481] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 669.741447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 669.874613] alg: No test for adler32 (adler32-zlib) [ 670.966470] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 671.271875] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 672.944226] Key type lgssc registered [ 674.095538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 682.418963] loop: module loaded [ 682.497854] loop0: detected capacity change from 0 to 204800000 [ 682.772602] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.787209] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.160585] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 689.555480] loop0: detected capacity change from 0 to 204800000 [ 689.701406] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 689.714770] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 689.736424] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 694.874271] loop0: detected capacity change from 0 to 204800000 [ 699.810356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 699.829505] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.040936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 701.093442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 701.178254] Lustre: lustre-MDT0000: new disk, initializing [ 701.250763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.286742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 706.032818] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 715.269367] loop1: detected capacity change from 0 to 204800000 [ 720.915733] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 721.325131] Lustre: lustre-OST07c6: new disk, initializing [ 721.329124] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 721.334118] Lustre: Skipped 1 previous similar message [ 721.347646] Lustre: 26176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 721.433756] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 727.893870] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 731.210316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 731.224387] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 731.256877] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 733.692957] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 733.703662] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 733.709219] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 733.714384] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 733.722177] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 733.730763] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 747.915250] Lustre: 26458:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 747.927329] Lustre: 26458:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 747.934217] Lustre: 26458:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 747.946734] Lustre: 26458:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 747.960591] Lustre: 26458:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 747.972396] Lustre: 26458:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 753.669474] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 753.678680] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 753.685706] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 753.689920] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.694328] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 753.698925] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.704737] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 753.715560] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.719870] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 753.723992] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.732040] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 753.739110] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.966356] Lustre: 26458:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 756.982522] Lustre: 26458:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 756.990969] Lustre: 26458:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 756.997179] Lustre: 26458:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.009695] Lustre: 26458:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 757.019387] Lustre: 26458:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.029643] Lustre: 26458:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 757.038501] Lustre: 26458:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.044480] Lustre: 26458:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 757.053366] Lustre: 26458:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.061526] Lustre: 26458:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 757.070127] Lustre: 26458:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 761.956078] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 761.975512] Lustre: 25163:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 761.981862] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 761.991948] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.002047] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.010575] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.019212] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 762.027364] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.035761] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 762.045231] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.053595] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.063204] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.966403] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 769.983822] Lustre: 25161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 769.995848] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 770.009272] Lustre: 25161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 770.027437] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 770.037060] Lustre: 25161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 770.048866] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 770.066849] Lustre: 25161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 770.080255] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 770.093483] Lustre: 25161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 770.112931] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 770.126899] Lustre: 25161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 828.387391] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 828.402258] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.418715] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 829.661607] Lustre: server umount lustre-OST07c6 complete [ 839.107111] Lustre: server umount lustre-MDT0000 complete [ 856.324648] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 859.019507] Key type lgssc unregistered [ 859.303267] LNet: 28730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.316448] LNetError: 28730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.346883] LNet: Removed LNI 192.168.206.115@tcp [ 860.345370] Key type .llcrypt unregistered [ 860.348562] Key type ._llcrypt unregistered [ 879.142387] Key type ._llcrypt registered [ 879.146967] Key type .llcrypt registered [ 879.271593] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 880.362046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 880.593795] alg: No test for adler32 (adler32-zlib) [ 881.715518] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 882.063066] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 883.801967] Key type lgssc registered [ 885.203282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 893.914253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 893.958773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.422309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.389320] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 905.702387] LustreError: 30000:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.727778] LustreError: 30000:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 909.638516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.060525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 915.017788] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 924.522615] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 931.480319] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 938.979922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 939.431493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.484485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 946.598620] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 954.887690] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 970.210872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 970.216636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.226185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.332774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.350368] Lustre: Skipped 1 previous similar message [ 975.683191] Lustre: server umount lustre-OST0000 complete [ 980.450903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 980.463461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 980.465804] Lustre: Skipped 2 previous similar messages [ 980.482141] Lustre: Skipped 2 previous similar messages [ 985.569791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 985.580246] Lustre: Skipped 1 previous similar message [ 985.671781] Lustre: server umount lustre-MDT0000 complete [ 989.390046] LustreError: 31851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438822 with bad export cookie 10605171640447230523 [ 989.398471] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 989.407699] LustreError: 31851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 989.687559] Lustre: server umount lustre-MDT0001 complete [ 999.786450] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1002.399602] Key type lgssc unregistered [ 1002.704627] LNet: 33742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.720379] LNetError: 33742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.738435] LNet: Removed LNI 192.168.206.115@tcp [ 1003.520785] Key type .llcrypt unregistered [ 1003.524207] Key type ._llcrypt unregistered [ 1020.939669] Key type ._llcrypt registered [ 1020.941973] Key type .llcrypt registered [ 1021.029917] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1022.064633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.178123] alg: No test for adler32 (adler32-zlib) [ 1023.223465] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 1023.456550] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1025.168222] Key type lgssc registered [ 1026.736477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1035.013627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1035.049570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.389805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1040.989196] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1047.015257] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1047.053530] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1051.445871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.085789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1058.457794] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1067.625968] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1074.469335] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1081.651873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1081.833239] LustreError: 36826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.049873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.471773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1088.609875] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1096.670113] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1106.916511] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1106.924569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1106.932520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1106.951491] Lustre: Skipped 1 previous similar message [ 1112.033845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1112.038168] Lustre: Skipped 1 previous similar message [ 1112.230352] Lustre: server umount lustre-OST0000 complete [ 1117.154684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.157288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1117.186243] Lustre: Skipped 1 previous similar message [ 1117.196514] Lustre: Skipped 1 previous similar message [ 1121.995383] Lustre: server umount lustre-MDT0000 complete [ 1122.279626] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.301223] LustreError: 35002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1126.078370] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789438958 with bad export cookie 3821766005873610595 [ 1126.098766] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1126.495864] Lustre: server umount lustre-MDT0001 complete [ 1137.859469] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1141.147257] Key type lgssc unregistered [ 1141.546749] LNet: 38536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.555392] LNetError: 38536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1141.606192] LNet: Removed LNI 192.168.206.115@tcp [ 1142.664273] Key type .llcrypt unregistered [ 1142.666282] Key type ._llcrypt unregistered [ 1166.802884] Key type ._llcrypt registered [ 1166.807384] Key type .llcrypt registered [ 1166.963618] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1167.851303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1167.885931] alg: No test for adler32 (adler32-zlib) [ 1168.942741] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 1169.159602] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1170.824277] Key type lgssc registered [ 1171.773376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1179.479107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1179.498195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.003182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.492044] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1190.373508] LustreError: 39950:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.402693] LustreError: 39950:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1194.112364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.577764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1199.390589] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1207.355246] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1213.844155] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1221.553543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1222.045380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.177548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1229.042129] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1236.913399] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1244.048747] Lustre: server umount lustre-MDT0000 complete [ 1244.618581] LustreError: 41964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.651515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.355750] LustreError: 39932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439081 with bad export cookie 14083778105162402427 [ 1248.359702] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1248.372700] LustreError: 39932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1249.693464] LustreError: 39944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.725675] LustreError: 39944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1249.736127] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.15@tcp (stopping) [ 1249.766185] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.776042] Lustre: Skipped 2 previous similar messages [ 1253.793374] LustreError: 41964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.796278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1253.824334] LustreError: 41964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1253.835980] Lustre: Skipped 1 previous similar message [ 1254.938804] Lustre: server umount lustre-MDT0001 complete [ 1276.651107] Lustre: server umount lustre-OST0000 complete [ 1296.337987] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1299.268587] Key type lgssc unregistered [ 1299.582703] LNet: 43810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.591656] LNetError: 43810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.607517] LNet: Removed LNI 192.168.206.115@tcp [ 1300.655353] Key type .llcrypt unregistered [ 1300.657234] Key type ._llcrypt unregistered [ 1327.202926] Key type ._llcrypt registered [ 1327.207279] Key type .llcrypt registered [ 1327.384429] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1328.616799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.861456] alg: No test for adler32 (adler32-zlib) [ 1330.335142] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 1330.650328] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1332.416241] Key type lgssc registered [ 1333.797894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1343.532672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1343.580389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.955145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1349.286918] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1354.220282] LustreError: 45224:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.263070] LustreError: 45224:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1358.376968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.746951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1362.734565] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1370.467589] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1376.705629] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1384.586494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1385.048405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.195329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1392.385377] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1399.664616] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1409.006476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1409.015012] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1409.031716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1409.038951] Lustre: Skipped 1 previous similar message [ 1410.124463] Lustre: server umount lustre-OST0000 complete [ 1429.095419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.099988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.113423] Lustre: Skipped 1 previous similar message [ 1429.122462] Lustre: Skipped 2 previous similar messages [ 1434.600942] LustreError: 45219:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.612660] LustreError: 45219:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1434.694773] Lustre: server umount lustre-MDT0000 complete [ 1438.289539] LustreError: 45204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439271 with bad export cookie 17222830304148503607 [ 1438.298221] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1438.308517] LustreError: 45204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1444.876429] Lustre: server umount lustre-MDT0001 complete [ 1456.594732] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1459.396918] Key type lgssc unregistered [ 1459.652266] LNet: 48932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.659853] LNetError: 48932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.674385] LNet: Removed LNI 192.168.206.115@tcp [ 1460.512421] Key type .llcrypt unregistered [ 1460.514742] Key type ._llcrypt unregistered [ 1482.957708] Key type ._llcrypt registered [ 1482.959692] Key type .llcrypt registered [ 1483.058975] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1483.864993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1483.959246] alg: No test for adler32 (adler32-zlib) [ 1484.997236] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 1485.233956] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1486.904193] Key type lgssc registered [ 1488.024838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1495.690866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1495.728625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.063717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.334868] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1506.284393] LustreError: 50336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.302218] LustreError: 50336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1508.901754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.318073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1513.420632] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1520.684757] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1526.468949] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1533.627368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1534.051373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.187447] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1541.244896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1547.928618] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1554.127118] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1558.195602] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1570.751667] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1422h30m02s [ 1572.630305] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1575.906113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1575.918626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1575.938552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1576.994512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1577.001737] Lustre: Skipped 5 previous similar messages [ 1581.468704] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 1581.481282] Lustre: Skipped 1 previous similar message [ 1587.612173] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 1587.621927] Lustre: Skipped 3 previous similar messages [ 1589.216178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1589.508583] Lustre: server umount lustre-MDT0000 complete [ 1593.113913] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1593.759824] LustreError: 50332:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.382182] LustreError: 50331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.395520] LustreError: 50331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1599.445834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1599.576831] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1599.884580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.973825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1600.921491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1604.449823] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1607.967949] Lustre: DEBUG MARKER: Wait for df (36659) ... [ 1609.613748] Lustre: DEBUG MARKER: done [ 1615.329248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1615.336137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.347284] Lustre: Skipped 2 previous similar messages [ 1615.355864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1615.364183] Lustre: Skipped 3 previous similar messages [ 1616.362924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.262953] Lustre: server umount lustre-OST0000 complete [ 1626.597479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1626.597596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.601332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1626.601340] Lustre: Skipped 2 previous similar messages [ 1626.636856] Lustre: Skipped 1 previous similar message [ 1637.344246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1637.505433] Lustre: server umount lustre-MDT0000 complete [ 1637.857713] LustreError: 50331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1637.874551] LustreError: 50331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1640.755517] LustreError: 50319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439473 with bad export cookie 2688769536912571783 [ 1640.759312] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1640.772920] LustreError: 50319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1641.087945] Lustre: server umount lustre-MDT0001 complete [ 1650.685703] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1653.251890] Key type lgssc unregistered [ 1653.527561] LNet: 55262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.538848] LNetError: 55262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.566691] LNet: Removed LNI 192.168.206.115@tcp [ 1654.343185] Key type .llcrypt unregistered [ 1654.345876] Key type ._llcrypt unregistered [ 1695.632520] Key type ._llcrypt registered [ 1695.641425] Key type .llcrypt registered [ 1695.784384] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1697.248858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1697.310448] alg: No test for adler32 (adler32-zlib) [ 1698.456449] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 1698.672436] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1700.360195] Key type lgssc registered [ 1701.390758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1709.268899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1709.293300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1710.615184] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1710.644260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1710.920606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1715.545421] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1723.086629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.198105] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1723.245644] Lustre: 57469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1723.514591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1727.529810] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1734.087650] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1739.563882] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1745.718106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1745.997720] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1746.275540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1752.208986] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1753.747266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1759.011160] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1765.673512] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1422h33m18s [ 1766.867847] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1773.771212] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1776.755073] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1776.823525] LustreError: 58218:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1797.079070] Lustre: lustre-MDT0000: Client 990cfd38-845e-44b1-9dc7-fe913bfe3d6d (at 192.168.206.15@tcp) reconnecting [ 1816.912111] LustreError: 58218:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1819.075425] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1821.684899] Lustre: DEBUG MARKER: done [ 1830.369092] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1830.382917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.395882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1830.887386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.895668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1830.895920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1832.871516] Lustre: server umount lustre-OST0000 complete [ 1839.015114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.023824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1839.027816] Lustre: Skipped 1 previous similar message [ 1842.595726] Lustre: server umount lustre-MDT0000 complete [ 1845.826398] LustreError: 57460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439678 with bad export cookie 15441363813584938653 [ 1845.835420] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1845.842419] LustreError: 57460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1846.157956] Lustre: server umount lustre-MDT0001 complete [ 1856.738417] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 1859.775760] Key type lgssc unregistered [ 1860.130160] LNet: 61796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1860.138208] LNetError: 61796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.153886] LNet: Removed LNI 192.168.206.115@tcp [ 1861.250792] Key type .llcrypt unregistered [ 1861.253851] Key type ._llcrypt unregistered [ 1899.685568] Key type ._llcrypt registered [ 1899.687919] Key type .llcrypt registered [ 1899.764657] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 1900.589299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1900.607967] alg: No test for adler32 (adler32-zlib) [ 1901.607914] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 1901.789486] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 1903.441367] Key type lgssc registered [ 1904.335898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1907.393431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1907.416151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.654547] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1908.675697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1908.802295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.613286] Lustre: Failing over lustre-MDT0000 [ 1909.856738] Lustre: server umount lustre-MDT0000 complete [ 1915.852588] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:35:48 (1789439748) [ 1921.034109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.320734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.902376] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1931.657297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.736935] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1931.767682] Lustre: 64570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1932.001129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1935.334285] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1941.549783] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1946.825960] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1952.178571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1952.314480] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1952.458855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1956.769661] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 1958.996751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1962.495236] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1967.670134] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1973.863891] Lustre: server umount lustre-MDT0000 complete [ 1974.248313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.260593] Lustre: Skipped 2 previous similar messages [ 1977.632287] LustreError: 65314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439810 with bad export cookie 12084811345412194472 [ 1977.655790] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1977.986847] Lustre: server umount lustre-MDT0001 complete [ 1981.312457] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1985.562031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1992.939673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.744134] Lustre: 63493:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789439812/real 1789439812] req@000000000a541ce5 x1876363558555136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789439828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1995.771109] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.880175] Lustre: 63493:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789439815/real 1789439815] req@00000000d2f3b63a x1876363558555264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789439831 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2000.864156] Lustre: 63495:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789439817/real 1789439817] req@000000001f237cb6 x1876363558555392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789439833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2002.976828] LustreError: 63491:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005dd8455f x1876363558556288/t0(0) o250->MGC192.168.206.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2003.402370] LustreError: 66420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.438301] LustreError: 66420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2003.534708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.664548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2004.577784] Lustre: 63495:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789439820/real 1789439820] req@00000000200ead8e x1876363558555648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789439836 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2005.604993] LustreError: 68339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.634987] LustreError: 68339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2009.017083] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2009.568920] Lustre: 63492:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789439825/real 1789439825] req@00000000b254c7a4 x1876363558555776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789439841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2011.109858] LustreError: 68337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2011.137674] LustreError: 68337:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2014.180281] LustreError: 68438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.181726] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2014.192588] LustreError: 68438:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2017.322414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.566639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.626222] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2022.893087] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2027.069569] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2032.755556] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2043.363308] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.366145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2043.366534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2043.389627] Lustre: Skipped 1 previous similar message [ 2045.383856] Lustre: server umount lustre-MDT0000 complete [ 2048.486049] LustreError: 68337:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2048.504049] LustreError: 68337:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2048.893258] LustreError: 64562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439881 with bad export cookie 12084811345412196754 [ 2048.907283] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2048.916643] LustreError: 64562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2049.209697] Lustre: server umount lustre-MDT0001 complete [ 2054.819053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2062.987832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.984169] Lustre: 63494:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789439886/real 1789439886] req@00000000fdbdc907 x1876363558581504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789439902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2070.011763] Lustre: 63494:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2070.019479] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.417367] LustreError: 69075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.487943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2074.547239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2078.012237] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2084.612304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.906488] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2089.977295] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2098.226858] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2098.427411] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2105.237413] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2113.506419] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.517819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2113.526071] Lustre: Skipped 1 previous similar message [ 2113.535574] Lustre: Skipped 3 previous similar messages [ 2118.628633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.633050] Lustre: Skipped 2 previous similar messages [ 2123.750558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.763222] Lustre: Skipped 1 previous similar message [ 2126.304121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2126.389989] Lustre: server umount lustre-MDT0000 complete [ 2128.869372] LustreError: 71022:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.880043] LustreError: 71022:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2129.461954] LustreError: 64562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789439962 with bad export cookie 12084811345412198133 [ 2129.470027] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2129.476566] LustreError: 64562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2129.654071] Lustre: server umount lustre-MDT0001 complete [ 2134.830678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2137.656475] Lustre: DEBUG MARKER: files compared the same [ 2149.729228] Lustre: server umount lustre-OST0000 complete [ 2163.639371] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2165.977868] Key type lgssc unregistered [ 2166.262790] LNet: 74298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2166.269754] LNetError: 74298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2166.295683] LNet: Removed LNI 192.168.206.115@tcp [ 2166.998553] Key type .llcrypt unregistered [ 2167.001842] Key type ._llcrypt unregistered [ 2188.733759] Key type ._llcrypt registered [ 2188.737533] Key type .llcrypt registered [ 2188.920229] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2190.379556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2190.410137] alg: No test for adler32 (adler32-zlib) [ 2191.483652] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 2191.806718] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2193.552391] Key type lgssc registered [ 2194.869694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2202.741423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2202.757639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2204.265081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2208.954563] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug +malloc all [ 2214.887329] LustreError: 75711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.914396] LustreError: 75711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2218.378264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.687685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2223.319174] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug +malloc all [ 2230.898822] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2236.447550] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2244.132455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2244.470282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2249.581046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2250.739623] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug +malloc all [ 2257.974316] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2274.284135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.287542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2274.317993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2274.322927] Lustre: Skipped 1 previous similar message [ 2276.745451] Lustre: server umount lustre-OST0000 complete [ 2284.514659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.517846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.536691] Lustre: Skipped 2 previous similar messages [ 2284.552913] Lustre: Skipped 1 previous similar message [ 2286.812613] Lustre: server umount lustre-MDT0000 complete [ 2289.637862] LustreError: 75711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.678535] LustreError: 75711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2290.683566] LustreError: 75691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789440123 with bad export cookie 11290968931186416857 [ 2290.691625] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2290.694467] LustreError: 75691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2291.059851] Lustre: server umount lustre-MDT0001 complete [ 2301.249434] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2303.837242] Key type lgssc unregistered [ 2304.178137] LNet: 79265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.184378] LNetError: 79265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.204717] LNet: Removed LNI 192.168.206.115@tcp [ 2305.268322] Key type .llcrypt unregistered [ 2305.278187] Key type ._llcrypt unregistered [ 2329.716631] Key type ._llcrypt registered [ 2329.719324] Key type .llcrypt registered [ 2329.851832] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2330.769326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2330.850604] alg: No test for adler32 (adler32-zlib) [ 2332.008490] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 2332.271549] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2334.081574] Key type lgssc registered [ 2335.593928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2343.474468] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2343.494422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2359.073272] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2364.192419] LustreError: 80653:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.115@tcp: failed processing log, type 4: rc = -110 [ 2390.880248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.362437] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2404.401147] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2412.328780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.406821] Lustre: *** cfs_fail_loc=706, val=0*** [ 2412.742962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.814063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2417.124995] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2418.147841] LustreError: 80678:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2418.168077] LustreError: 80678:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2423.266828] LustreError: 80679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.907595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2426.274600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2430.465458] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2437.964871] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2443.448250] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2446.825611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2446.833454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.843085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2447.845354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2447.847375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.522086] Lustre: server umount lustre-OST0000 complete [ 2457.057603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2457.071994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.090348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2457.095759] Lustre: Skipped 1 previous similar message [ 2461.696051] Lustre: server umount lustre-MDT0000 complete [ 2463.208085] LustreError: 82547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.230449] LustreError: 82547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2465.827496] LustreError: 82481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789440298 with bad export cookie 8249602135454133358 [ 2465.832491] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2465.843535] LustreError: 82481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2466.305482] Lustre: server umount lustre-MDT0001 complete [ 2477.439679] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2480.386987] Key type lgssc unregistered [ 2480.660101] LNet: 84268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2480.675299] LNetError: 84268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2480.699277] LNet: Removed LNI 192.168.206.115@tcp [ 2481.868380] Key type .llcrypt unregistered [ 2481.871073] Key type ._llcrypt unregistered [ 2509.899489] Key type ._llcrypt registered [ 2509.907889] Key type .llcrypt registered [ 2510.101256] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2511.403174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2511.597062] alg: No test for adler32 (adler32-zlib) [ 2512.752365] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 2513.056283] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2514.784253] Key type lgssc registered [ 2515.982569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2523.899501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2523.929875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.964886] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2538.648336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.318858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2544.594901] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2544.614972] LustreError: 86369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.632909] LustreError: 86369:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2549.730952] LustreError: 86372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.748836] LustreError: 86372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2553.053769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2553.422149] LustreError: 86369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.446011] LustreError: 86369:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2553.501644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.948351] LustreError: 86367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.963168] LustreError: 86367:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2559.934081] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2564.071162] LustreError: 86373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.103599] LustreError: 86373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2569.009946] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2569.034400] Lustre: MGS: Client e1e6a154-954d-4574-bba4-3c08b71dba53 (at 0@lo) reconnecting [ 2569.057625] LustreError: 86383:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2569.384765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.488959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2574.768265] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2590.181304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2590.188859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.190143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2590.220146] Lustre: Skipped 1 previous similar message [ 2594.623139] Lustre: server umount lustre-OST0000 complete [ 2600.419307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2600.420763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.430691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2600.442108] Lustre: Skipped 1 previous similar message [ 2600.451444] Lustre: Skipped 2 previous similar messages [ 2605.540148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2605.548703] Lustre: Skipped 1 previous similar message [ 2610.661917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2612.192631] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2612.298781] Lustre: server umount lustre-MDT0000 complete [ 2615.783506] LustreError: 86373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.806112] LustreError: 86373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2616.177428] Lustre: server umount lustre-MDT0001 complete [ 2619.919241] Lustre: server umount MGS complete [ 2632.678064] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:47:45 (1789440465) [ 2663.982876] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 2672.867846] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2708.466332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.350200] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2720.877902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2721.202767] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2721.256683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2721.360456] Lustre: lustre-MDT0001: new disk, initializing [ 2721.451261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2721.490033] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2721.503767] Lustre: 93539:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2721.512047] LustreError: 93539:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2725.841148] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2735.632624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.000138] Lustre: lustre-OST0000: new disk, initializing [ 2736.009329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2736.020244] Lustre: 94305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2736.111360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.479819] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2746.405291] Lustre: 94766:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b81891f9 x1876364199240576/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789440595 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2751.509404] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2757.566892] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2757.589067] Lustre: MGS: Client 5f500c4e-0eb7-48a0-a93b-a13d70db8286 (at 0@lo) reconnecting [ 2757.643700] Lustre: 95350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2757.660370] LustreError: 93525:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2757.667377] LustreError: 93525:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2757.685075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2757.825849] Lustre: lustre-MDT0000: new disk, initializing [ 2757.974403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2757.994504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2758.121543] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2759.078963] Lustre: 95374:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ee94ab14 x1876364199246976/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789440607 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2762.392273] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2766.844101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2766.861986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2768.182779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2768.195584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2768.240585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2790.376435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.378382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2790.391279] Lustre: Skipped 1 previous similar message [ 2790.400741] Lustre: Skipped 2 previous similar messages [ 2796.707759] Lustre: server umount lustre-OST0000 complete [ 2800.610364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2800.621338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.655553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.660121] Lustre: Skipped 3 previous similar messages [ 2806.763769] Lustre: server umount lustre-MDT0000 complete [ 2816.576089] Lustre: server umount lustre-MDT0001 complete [ 2820.411877] Lustre: server umount MGS complete [ 2832.912515] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:51:05 (1789440665) [ 2851.159674] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 2853.745507] Key type lgssc unregistered [ 2854.183856] LNet: 97870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2854.195205] LNetError: 97870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2854.255640] LNet: Removed LNI 192.168.206.115@tcp [ 2855.390579] Key type .llcrypt unregistered [ 2855.395622] Key type ._llcrypt unregistered [ 2877.948505] Key type ._llcrypt registered [ 2877.951216] Key type .llcrypt registered [ 2878.136696] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 2879.380096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2879.413379] alg: No test for adler32 (adler32-zlib) [ 2880.626523] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 2880.820855] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 2882.496890] Key type lgssc registered [ 2883.812677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2891.911395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2891.939600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.375843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.967449] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2903.529657] LustreError: 99088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.566032] LustreError: 99088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2906.600812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.972532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2911.780974] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2919.794964] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2925.958781] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2933.503889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.798874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2938.927141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2939.568498] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 2946.774858] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2959.329056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2959.334935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.350075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.383935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2961.384649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.394231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.408636] Lustre: Skipped 1 previous similar message [ 2964.253353] Lustre: server umount lustre-OST0000 complete [ 2968.545320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2968.560229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2968.593304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2971.622435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2971.631132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2976.739442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2976.754183] Lustre: Skipped 2 previous similar messages [ 2982.369100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2982.582351] Lustre: server umount lustre-MDT0000 complete [ 2986.979083] LustreError: 99088:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2987.003018] LustreError: 99088:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2987.033079] LustreError: 99070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789440819 with bad export cookie 901704063321875970 [ 2987.040309] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2987.048494] LustreError: 99070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2987.508586] Lustre: server umount lustre-MDT0001 complete [ 2998.674573] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3001.418290] Key type lgssc unregistered [ 3001.740344] LNet: 102620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3001.750868] LNetError: 102620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3001.770806] LNet: Removed LNI 192.168.206.115@tcp [ 3002.858890] Key type .llcrypt unregistered [ 3002.862941] Key type ._llcrypt unregistered [ 3021.247810] Key type ._llcrypt registered [ 3021.252404] Key type .llcrypt registered [ 3021.399861] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3022.694117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3022.853741] alg: No test for adler32 (adler32-zlib) [ 3024.166202] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 3024.645136] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3026.411302] Key type lgssc registered [ 3028.264988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3034.099588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3034.100033] LustreError: 103700:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3034.121148] Lustre: Skipped 1 previous similar message [ 3034.134350] LustreError: 103701:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3034.142670] LustreError: 103700:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3034.149487] LustreError: 103700:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3034.162897] LustreError: 103700:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3034.173594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.764238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3044.025313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.269443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3049.050138] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3054.837103] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing lsmod [ 3057.753537] LustreError: 104906:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3057.754525] LustreError: 104905:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3057.768485] LustreError: 104906:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 3057.780038] LustreError: 104906:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3057.791885] LustreError: 104906:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3057.801597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3058.165531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.219822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3064.802329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3064.810387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.837961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.315630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.327295] Lustre: Skipped 1 previous similar message [ 3065.339537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.349790] Lustre: Skipped 2 previous similar messages [ 3067.984097] Lustre: server umount lustre-MDT0000 complete [ 3069.482939] LustreError: 103737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.517237] LustreError: 103737:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3072.283789] LustreError: 103719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789440905 with bad export cookie 3890453317707957640 [ 3072.304887] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3073.632726] LustreError: 104169:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000003796ffeb x1876364736091648/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3073.824982] Lustre: server umount lustre-MDT0001 complete [ 3088.128832] Lustre: server umount lustre-OST0000 complete [ 3098.118961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3098.585540] LustreError: 105972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.681816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.714329] LustreError: 105973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.247427] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3108.835376] LustreError: 105973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.834034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.191907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3117.766519] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3126.518219] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3132.658776] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3140.484703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3140.977111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.060877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3147.789157] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3155.396207] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3167.719381] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.725130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3167.730101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3167.735229] Lustre: Skipped 1 previous similar message [ 3167.758865] Lustre: Skipped 1 previous similar message [ 3172.848456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3172.856064] Lustre: Skipped 1 previous similar message [ 3173.108826] Lustre: server umount lustre-OST0000 complete [ 3177.954179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.961099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3177.973787] Lustre: Skipped 1 previous similar message [ 3177.992881] Lustre: Skipped 1 previous similar message [ 3188.198295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.206149] Lustre: Skipped 3 previous similar messages [ 3191.264148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3191.423211] Lustre: server umount lustre-MDT0000 complete [ 3193.320800] LustreError: 105973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.342781] LustreError: 105973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3195.012892] LustreError: 106706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789441027 with bad export cookie 3890453317707958585 [ 3195.014894] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3195.023428] LustreError: 106706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3195.327986] Lustre: server umount lustre-MDT0001 complete [ 3207.264786] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3210.142318] Key type lgssc unregistered [ 3210.475559] LNet: 109508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3210.484415] LNetError: 109508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3210.504799] LNet: Removed LNI 192.168.206.115@tcp [ 3211.383763] Key type .llcrypt unregistered [ 3211.385917] Key type ._llcrypt unregistered [ 3232.064196] Key type ._llcrypt registered [ 3232.068271] Key type .llcrypt registered [ 3232.154419] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3232.682746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3232.702541] alg: No test for adler32 (adler32-zlib) [ 3233.787416] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 3234.069744] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3235.760251] Key type lgssc registered [ 3237.124796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3244.527543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3244.539059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.994835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.363741] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3255.277103] LustreError: 110918:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.312087] LustreError: 110918:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3260.404255] LustreError: 110919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3261.237419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3261.562653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3265.496491] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3272.304474] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3278.217773] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3284.397530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3284.711229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3290.878359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3291.305290] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3299.070061] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3305.816379] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3309.610363] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3313.637301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3313.649389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.679387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3315.297259] Lustre: server umount lustre-OST0000 complete [ 3321.826541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.835120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3321.849141] Lustre: Skipped 1 previous similar message [ 3321.856334] Lustre: Skipped 1 previous similar message [ 3326.947251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3326.965242] Lustre: Skipped 1 previous similar message [ 3330.179972] Lustre: server umount lustre-MDT0000 complete [ 3332.070395] LustreError: 110919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.095881] LustreError: 110919:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3334.273653] LustreError: 110898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789441167 with bad export cookie 7634139716361761565 [ 3334.279320] LustreError: 110898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3334.283788] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3334.632265] Lustre: server umount lustre-MDT0001 complete [ 3346.450278] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3349.337774] Key type lgssc unregistered [ 3349.674855] LNet: 114556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3349.692076] LNetError: 114556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3349.721551] LNet: Removed LNI 192.168.206.115@tcp [ 3350.718441] Key type .llcrypt unregistered [ 3350.722146] Key type ._llcrypt unregistered [ 3367.604698] Key type ._llcrypt registered [ 3367.607026] Key type .llcrypt registered [ 3367.760640] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3368.908074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3368.929662] alg: No test for adler32 (adler32-zlib) [ 3369.970283] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 3370.156454] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3371.808224] Key type lgssc registered [ 3372.584832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3379.136654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3379.158896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.694645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.744692] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3390.946832] LustreError: 115772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.975391] LustreError: 115772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3392.540507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.892390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3397.059840] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3405.256170] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3411.706538] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3420.071147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.382293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.571365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3426.572458] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3434.789974] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3451.361451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3451.374273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.397693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3453.410370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3453.413406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.414619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3456.183841] Lustre: server umount lustre-OST0000 complete [ 3463.649386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.652396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3463.665337] Lustre: Skipped 1 previous similar message [ 3463.667222] Lustre: Skipped 1 previous similar message [ 3467.130002] Lustre: server umount lustre-MDT0000 complete [ 3468.776610] LustreError: 118321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3468.804487] LustreError: 118321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3471.170093] LustreError: 115754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789441304 with bad export cookie 8312855543064231398 [ 3471.189048] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3471.620399] Lustre: server umount lustre-MDT0001 complete [ 3483.106523] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3485.484879] Key type lgssc unregistered [ 3485.756122] LNet: 119343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3485.772540] LNetError: 119343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3485.790723] LNet: Removed LNI 192.168.206.115@tcp [ 3486.636244] Key type .llcrypt unregistered [ 3486.640964] Key type ._llcrypt unregistered [ 3509.981136] Key type ._llcrypt registered [ 3509.985191] Key type .llcrypt registered [ 3510.151892] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3511.242975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3511.287720] alg: No test for adler32 (adler32-zlib) [ 3512.359886] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 3512.514402] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3514.240191] Key type lgssc registered [ 3515.755267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3524.671489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3524.684285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.283614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3531.273710] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3536.867641] LustreError: 120773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.877762] LustreError: 120773:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3539.926593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.263319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.955151] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3553.262939] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3560.097642] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3567.348623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3567.739041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3571.854592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3574.274470] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3581.623458] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3592.602913] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3596.865744] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3596.870482] Lustre: Skipped 2 previous similar messages [ 3600.985852] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3601.031655] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3601.533549] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3601.543076] Lustre: Skipped 2 previous similar messages [ 3607.641303] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3607.649551] Lustre: Skipped 1 previous similar message [ 3613.149623] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3613.172609] Lustre: Skipped 2 previous similar messages [ 3617.215482] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.15@tcp 192.168.0.[2,10]@tcp [ 3622.307509] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.15/24@tcp in log lustre-MDT0000 [ 3622.312751] Lustre: Skipped 2 previous similar messages [ 3624.370815] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.15/24@tcp [ 3624.384483] Lustre: Skipped 1 previous similar message [ 3635.177720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3635.184805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3635.218046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3636.667488] Lustre: server umount lustre-OST0000 complete [ 3641.828819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.843925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.952967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.962243] Lustre: Skipped 2 previous similar messages [ 3647.212174] Lustre: server umount lustre-MDT0000 complete [ 3651.040957] LustreError: 123810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789441483 with bad export cookie 10636743915730488917 [ 3651.042867] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.054106] LustreError: 123810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3660.502433] Lustre: server umount lustre-MDT0001 complete [ 3673.878328] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3677.158900] Key type lgssc unregistered [ 3677.520599] LNet: 125624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3677.541608] LNetError: 125624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3677.556192] LNet: Removed LNI 192.168.206.115@tcp [ 3678.503792] Key type .llcrypt unregistered [ 3678.509237] Key type ._llcrypt unregistered [ 3710.561752] Key type ._llcrypt registered [ 3710.564319] Key type .llcrypt registered [ 3710.690185] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3711.637660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3711.664868] alg: No test for adler32 (adler32-zlib) [ 3712.705910] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 3712.977757] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3714.640255] Key type lgssc registered [ 3715.901302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3724.117929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3724.155873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3725.670423] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3725.717062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.722733] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.15/24@tcp [ 3730.785372] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3736.036030] LustreError: 127186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.058961] LustreError: 127186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3739.156780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3739.540968] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3739.573723] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.15/24@tcp [ 3739.627602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3743.999856] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3751.104567] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3757.062969] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3764.319659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3764.721123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3770.870779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3771.222348] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3779.547862] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3791.334195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3791.337021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.344893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3791.383304] Lustre: Skipped 1 previous similar message [ 3796.024059] Lustre: server umount lustre-OST0000 complete [ 3801.061457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3801.066067] LustreError: Skipped 1 previous similar message [ 3801.075781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.100903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3801.107050] Lustre: Skipped 1 previous similar message [ 3806.122453] Lustre: server umount lustre-MDT0000 complete [ 3806.694558] LustreError: 127186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3806.716604] LustreError: 127186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3809.855292] LustreError: 127165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789441642 with bad export cookie 11862309046553162357 [ 3809.856320] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3809.872321] LustreError: 127165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3810.155488] Lustre: server umount lustre-MDT0001 complete [ 3821.409956] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 3824.292277] Key type lgssc unregistered [ 3824.591452] LNet: 130774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3824.596407] LNetError: 130774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3824.630816] LNet: Removed LNI 192.168.206.115@tcp [ 3825.463589] Key type .llcrypt unregistered [ 3825.465581] Key type ._llcrypt unregistered [ 3849.238528] Key type ._llcrypt registered [ 3849.240647] Key type .llcrypt registered [ 3849.338732] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 3863.174190] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 3863.841283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3863.918221] alg: No test for adler32 (adler32-zlib) [ 3864.881881] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 3865.136726] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 3866.768338] Key type lgssc registered [ 3867.745768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3898.719883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3898.780243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.002505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3900.028885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3900.090177] Lustre: lustre-MDT0000: new disk, initializing [ 3900.145964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.158561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3904.351180] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3914.942466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.106228] Lustre: 134184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3915.186646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3915.190489] Lustre: Skipped 1 previous similar message [ 3915.374298] Lustre: lustre-MDT0001: new disk, initializing [ 3915.515424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3915.563644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3915.577158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3919.933664] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3929.297474] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3934.951090] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3942.677724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3943.048677] Lustre: lustre-OST0000: new disk, initializing [ 3943.055764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3943.078498] Lustre: 136154:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3943.216614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3949.937316] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 3952.728454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3952.746170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3952.856862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3960.458818] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3965.039125] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3965.275558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3969.630543] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3969.865609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3973.602648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3973.617788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.628738] LustreError: Skipped 1 previous similar message [ 3973.633305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3973.656840] Lustre: Skipped 1 previous similar message [ 3973.678690] Lustre: Skipped 1 previous similar message [ 3977.911990] Lustre: server umount lustre-OST0000 complete [ 3982.308058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3982.317860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.339724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3983.842435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.843666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3983.871065] Lustre: Skipped 1 previous similar message [ 3988.964798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3988.971949] Lustre: Skipped 1 previous similar message [ 3994.086238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3994.092749] Lustre: Skipped 1 previous similar message [ 3996.128260] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3996.342949] Lustre: server umount lustre-MDT0000 complete [ 4000.525610] LustreError: 134174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789441833 with bad export cookie 4369478645143294648 [ 4000.527922] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4000.538030] LustreError: 134174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4000.937489] Lustre: server umount lustre-MDT0001 complete [ 4008.767294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4009.372283] LustreError: 138372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4009.400414] LustreError: 138372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4009.490931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4014.000505] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4014.568921] LustreError: 138373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4019.690337] LustreError: 138372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4022.639646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.068307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4028.196589] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4035.750092] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4040.940412] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4047.824478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4048.154330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4055.611931] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4064.700433] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4068.999784] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4069.211427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4074.329318] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4074.525387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4085.054160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4085.241140] Lustre: lustre-OST0001: new disk, initializing [ 4085.244909] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4085.250537] Lustre: 141744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4085.338801] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4092.394579] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4094.578201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4094.587918] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4094.678272] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4102.337661] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4102.612898] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4106.912845] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4107.064436] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4113.164128] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e099193d000.ost_server_uuid 50 [ 4115.344641] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e099193d000.ost_server_uuid in IDLE state after 0 sec [ 4130.794102] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.803540] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4130.820413] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4130.824826] Lustre: Skipped 1 previous similar message [ 4130.849852] LustreError: Skipped 1 previous similar message [ 4132.658757] Lustre: server umount lustre-OST0001 complete [ 4135.910945] LustreError: 140193:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4135.944128] LustreError: 140193:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4139.491644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4139.508111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.034761] LustreError: 141295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.057106] LustreError: 141295:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4143.825301] Lustre: server umount lustre-OST0000 complete [ 4151.268504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.282212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4151.287485] Lustre: Skipped 1 previous similar message [ 4151.290129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4151.308897] Lustre: Skipped 5 previous similar messages [ 4154.251700] Lustre: server umount lustre-MDT0000 complete [ 4156.385587] LustreError: 143036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.423329] LustreError: 143036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4158.282363] LustreError: 138352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789441991 with bad export cookie 4369478645143295530 [ 4158.293275] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4158.297311] LustreError: 138352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4158.722189] Lustre: server umount lustre-MDT0001 complete [ 4181.044132] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4185.101553] Key type lgssc unregistered [ 4185.630448] LNet: 144746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4185.662651] LNetError: 144746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4185.686956] LNet: Removed LNI 192.168.206.115@tcp [ 4186.856714] Key type .llcrypt unregistered [ 4186.859093] Key type ._llcrypt unregistered [ 4231.657510] Key type ._llcrypt registered [ 4231.660498] Key type .llcrypt registered [ 4231.803575] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4248.418592] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4249.992228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4250.079090] alg: No test for adler32 (adler32-zlib) [ 4251.097916] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 4251.326888] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4253.041684] Key type lgssc registered [ 4254.667324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4293.021076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4293.069752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.430386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4294.451579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4294.509905] Lustre: lustre-MDT0000: new disk, initializing [ 4294.588395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.617397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4300.620795] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4312.133690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.232878] Lustre: 148899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4312.280386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4312.287082] Lustre: Skipped 1 previous similar message [ 4312.382677] Lustre: lustre-MDT0001: new disk, initializing [ 4312.508181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4312.565186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4312.578348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4318.129332] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4328.340257] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4334.161254] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4341.187497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4341.430730] Lustre: lustre-OST0000: new disk, initializing [ 4341.435598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4341.441435] Lustre: 150865:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4341.567347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4347.964510] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4351.610412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4351.628520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4351.743507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4357.944830] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4363.655833] Lustre: Failing over lustre-OST0000 [ 4363.928559] Lustre: server umount lustre-OST0000 complete [ 4380.641149] Lustre: 147120:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789442197/real 1789442197] req@000000003356e46d x1876366022705792/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789442213 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4380.668559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.463673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4383.712566] Lustre: 147119:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789442200/real 1789442200] req@0000000023a43dfd x1876366022706048/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789442216 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4383.735664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.830497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4383.850927] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4384.865185] Lustre: 147120:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789442200/real 1789442200] req@00000000256eea76 x1876366022706816/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789442216 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4384.893898] Lustre: 147120:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4385.854264] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4388.833811] Lustre: 147120:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789442205/real 1789442205] req@00000000c88966d3 x1876366022707840/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789442221 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4391.437685] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4392.405388] Lustre: lustre-OST0000: Recovery over after 0:07, of 3 clients 3 recovered and 0 were evicted. [ 4392.407606] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4393.504125] Lustre: 147120:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789442210/real 1789442210] req@0000000023571600 x1876366022708992/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789442226 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4393.532451] Lustre: 147120:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4395.814233] Lustre: Failing over lustre-MDT0000 [ 4396.237155] Lustre: server umount lustre-MDT0000 complete [ 4396.524698] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.550854] Lustre: Skipped 2 previous similar messages [ 4401.634821] LustreError: 148910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4401.663401] LustreError: 148910:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4402.590298] LustreError: 148904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.758653] LustreError: 148905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.782373] LustreError: 148905:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4411.875985] LustreError: 148909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4411.896030] LustreError: 148909:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4412.899183] Lustre: 147118:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789442229/real 1789442229] req@00000000665459a7 x1876366022720000/t0(0) o400->MGC192.168.206.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789442245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4412.929945] Lustre: 147118:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4412.942434] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4415.597828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4417.949153] LustreError: 150266:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.969102] LustreError: 150266:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4423.146669] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc83f42a24016020d [ 4423.164465] Lustre: MGC192.168.206.115@tcp: Connection restored to 0@lo (at 0@lo) [ 4423.173884] Lustre: Skipped 1 previous similar message [ 4423.454097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.543520] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4424.090659] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4428.414582] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4428.806107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4428.845554] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4428.907097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4438.497715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4438.505546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4438.526914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4444.133076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4444.141438] Lustre: Skipped 2 previous similar messages [ 4449.256664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4449.263060] Lustre: Skipped 1 previous similar message [ 4451.297837] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4451.600077] Lustre: server umount lustre-OST0000 complete [ 4459.496847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4459.498870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4459.526452] Lustre: Skipped 1 previous similar message [ 4459.534151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4459.539830] Lustre: Skipped 2 previous similar messages [ 4462.042705] Lustre: server umount lustre-MDT0000 complete [ 4464.609069] LustreError: 148909:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.622768] LustreError: 148909:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 4466.507138] LustreError: 148890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789442299 with bad export cookie 14429324995745677837 [ 4466.523901] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4466.529175] LustreError: 148890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4467.083960] Lustre: server umount lustre-MDT0001 complete [ 4479.026806] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 4482.089571] Key type lgssc unregistered [ 4482.504335] LNet: 154532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4482.522533] LNetError: 154532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4482.563459] LNet: Removed LNI 192.168.206.115@tcp [ 4483.737375] Key type .llcrypt unregistered [ 4483.743683] Key type ._llcrypt unregistered [ 4523.958129] Key type ._llcrypt registered [ 4523.962320] Key type .llcrypt registered [ 4524.093411] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4525.143442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4525.183918] alg: No test for adler32 (adler32-zlib) [ 4526.440572] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 4526.685163] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 4528.376760] Key type lgssc registered [ 4529.662976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4539.610052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4539.645633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4541.326980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4546.300853] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4551.649446] LustreError: 156050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4551.677636] LustreError: 156050:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4556.777853] LustreError: 156051:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4557.509448] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4561.902994] LustreError: 156050:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.291322] LustreError: 156079:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4562.308626] LustreError: 156079:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4562.320463] LustreError: 156079:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4562.806212] Lustre: server umount lustre-MDT0000 complete [ 4579.644762] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 4590.044982] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 4626.348509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4626.617784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4626.676884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4626.801598] Lustre: lustre-MDT0000: new disk, initializing [ 4626.893080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4626.913748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4631.620493] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4642.620429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.813738] Lustre: 159385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4642.876256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4642.887266] Lustre: Skipped 1 previous similar message [ 4642.976675] Lustre: lustre-MDT0001: new disk, initializing [ 4643.119961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4643.179284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4643.195827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4648.573321] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4659.067525] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4666.406235] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4673.489041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4673.800671] Lustre: lustre-OST0000: new disk, initializing [ 4673.803809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4673.809480] Lustre: 161354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4673.904304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4680.620245] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 4682.938141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4682.962536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4683.181809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4690.884749] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4694.837601] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4696.939625] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:22:09 (1789442529) [ 4954.593058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4954.595668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4954.620496] Lustre: Skipped 1 previous similar message [ 4954.640685] Lustre: Skipped 2 previous similar messages [ 4959.713138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4959.722148] Lustre: Skipped 2 previous similar messages [ 4964.837335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4964.840415] Lustre: Skipped 2 previous similar messages [ 4967.392121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4967.627816] Lustre: server umount lustre-MDT0000 complete [ 4969.953690] LustreError: 159396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4969.989356] LustreError: 159396:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4971.560767] LustreError: 159377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789442804 with bad export cookie 2443389040323216696 [ 4971.562878] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4971.587264] LustreError: 159377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4975.073590] LustreError: 163121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4975.074990] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4975.119380] Lustre: Skipped 1 previous similar message [ 4975.124330] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4978.068615] Lustre: server umount lustre-MDT0001 complete [ 4988.180144] Lustre: server umount lustre-OST0000 complete [ 4998.548561] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 5009.704345] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5051.243991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5051.437398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5051.464415] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5051.535945] Lustre: lustre-MDT0000: new disk, initializing [ 5051.607971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5051.639126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5056.035671] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5067.027273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5067.137294] Lustre: 166020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5067.183960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5067.188590] Lustre: Skipped 1 previous similar message [ 5067.254590] Lustre: lustre-MDT0001: new disk, initializing [ 5067.334161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5067.367729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5067.379223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5072.557897] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5082.287750] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5088.498588] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5096.409779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5096.655722] Lustre: lustre-OST0000: new disk, initializing [ 5096.660604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5096.667405] Lustre: 167989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5096.766205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5102.596522] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5104.186533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5104.197203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5104.312649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5113.575219] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5121.367848] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5121.626792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5128.251541] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5128.556340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5135.342414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5135.353267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5135.392544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5135.396251] Lustre: Skipped 1 previous similar message [ 5137.414823] Lustre: server umount lustre-OST0000 complete [ 5144.034141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5144.066712] LustreError: Skipped 1 previous similar message [ 5144.074060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.093287] Lustre: Skipped 1 previous similar message [ 5144.114203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5156.320117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5156.787711] Lustre: server umount lustre-MDT0000 complete [ 5160.935450] LustreError: 167212:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.961415] LustreError: 167212:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5161.387568] LustreError: 166012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789442994 with bad export cookie 2443389040323375519 [ 5161.401280] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.827680] Lustre: server umount lustre-MDT0001 complete [ 5171.548365] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:30:03 (1789443003) [ 5188.770517] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 5199.897466] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5236.602901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.776415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5236.804918] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5236.880803] Lustre: lustre-MDT0000: new disk, initializing [ 5236.963992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5236.983540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5241.242171] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5251.866093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.000449] Lustre: 172747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5252.289608] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5256.523293] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5266.031456] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5272.073901] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5279.525221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5279.949911] Lustre: lustre-OST0000: new disk, initializing [ 5279.954912] Lustre: Skipped 1 previous similar message [ 5279.963091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5279.968116] Lustre: Skipped 2 previous similar messages [ 5279.975272] Lustre: 174723:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5286.838338] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5289.014394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5289.020707] Lustre: Skipped 1 previous similar message [ 5289.031616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5289.190663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5296.115366] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5304.801402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5304.813431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5304.819371] LustreError: Skipped 1 previous similar message [ 5304.827543] Lustre: Skipped 1 previous similar message [ 5304.859792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5304.865497] Lustre: Skipped 6 previous similar messages [ 5309.171487] Lustre: server umount lustre-OST0000 complete [ 5314.017553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5314.028169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5314.043381] Lustre: Skipped 1 previous similar message [ 5319.801591] Lustre: server umount lustre-MDT0000 complete [ 5320.172240] LustreError: 172758:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5320.198579] LustreError: 172758:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5323.514930] LustreError: 174722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789443156 with bad export cookie 2443389040323376408 [ 5323.518074] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5323.530282] LustreError: 174722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5324.020921] Lustre: server umount lustre-MDT0001 complete [ 5332.853188] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:32:45 (1789443165) [ 5347.842969] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 5358.393326] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5396.544566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5396.774736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5396.795802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5396.890107] Lustre: lustre-MDT0000: new disk, initializing [ 5396.994812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.007707] Lustre: Skipped 2 previous similar messages [ 5397.039872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5401.268965] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5410.793421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.902664] Lustre: 179046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5411.160779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5415.302724] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5424.370353] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5430.294688] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5437.027215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5437.234466] Lustre: 181021:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5443.686876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5443.855522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5445.409186] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5454.989398] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5464.547263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5464.554605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5464.558608] Lustre: Skipped 2 previous similar messages [ 5464.571524] Lustre: Skipped 5 previous similar messages [ 5469.304908] Lustre: server umount lustre-OST0000 complete [ 5479.485975] Lustre: server umount lustre-MDT0000 complete [ 5479.908795] LustreError: 181241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5479.930853] LustreError: 181241:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5482.988643] LustreError: 179040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789443315 with bad export cookie 2443389040323377493 [ 5482.995834] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5483.002619] LustreError: 179040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5483.302322] Lustre: server umount lustre-MDT0001 complete [ 5495.329043] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5498.557253] Key type lgssc unregistered [ 5498.843421] LNet: 182876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5498.854202] LNetError: 182876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5498.883224] LNet: Removed LNI 192.168.206.115@tcp [ 5499.748150] Key type .llcrypt unregistered [ 5499.749742] Key type ._llcrypt unregistered [ 5522.928645] Key type ._llcrypt registered [ 5522.930696] Key type .llcrypt registered [ 5523.021925] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5523.800652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5523.820419] alg: No test for adler32 (adler32-zlib) [ 5524.786805] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 5524.919616] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5526.560225] Key type lgssc registered [ 5527.504492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5535.356952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5535.389412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.967851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5541.799854] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5546.466691] LustreError: 184286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5546.497652] LustreError: 184286:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5550.126370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5550.527752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5555.032504] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5561.656505] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5566.936993] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5572.856784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5573.162626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5578.467701] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5581.374867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5584.518739] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5605.862591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5605.880234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.896880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5606.886254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.900733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5606.913560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5606.921665] Lustre: Skipped 1 previous similar message [ 5608.930497] Lustre: server umount lustre-OST0000 complete [ 5615.022182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5615.026675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5615.043452] Lustre: Skipped 1 previous similar message [ 5615.060504] Lustre: Skipped 1 previous similar message [ 5619.486433] Lustre: server umount lustre-MDT0000 complete [ 5621.218181] LustreError: 185016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5621.233201] LustreError: 185016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5623.692657] LustreError: 184267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789443456 with bad export cookie 3140068755967672955 [ 5623.699369] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5623.706262] LustreError: 184267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5624.165050] Lustre: server umount lustre-MDT0001 complete [ 5634.914937] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 5637.469183] Key type lgssc unregistered [ 5637.767415] LNet: 187838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5637.770639] LNetError: 187838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5637.790260] LNet: Removed LNI 192.168.206.115@tcp [ 5638.698237] Key type .llcrypt unregistered [ 5638.706766] Key type ._llcrypt unregistered [ 5664.604183] Key type ._llcrypt registered [ 5664.610060] Key type .llcrypt registered [ 5664.792882] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 5665.963387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5666.073194] alg: No test for adler32 (adler32-zlib) [ 5667.083709] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 5667.342067] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 5669.008178] Key type lgssc registered [ 5670.212159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5678.628958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5678.662505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.157269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5684.305351] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5689.316054] LustreError: 189256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5689.336391] LustreError: 189256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5692.827862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5693.220322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5697.898481] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5705.502564] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5711.366591] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5718.238608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5718.729236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.908664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5725.003604] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5733.133621] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5745.595774] Lustre: lustre-OST0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 5745.640332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5745.647618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.690527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5747.694455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5747.698305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.729587] Lustre: Skipped 2 previous similar messages [ 5748.775512] Lustre: server umount lustre-OST0000 complete [ 5756.073577] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8e0984c01800.ost_server_uuid 50 [ 5758.089436] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5771.313882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.328470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5771.341704] Lustre: Skipped 1 previous similar message [ 5773.350336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5773.355212] Lustre: Skipped 1 previous similar message [ 5777.600050] Lustre: server umount lustre-MDT0000 complete [ 5778.401661] LustreError: 189253:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.415626] LustreError: 189253:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5781.732972] LustreError: 189238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789443614 with bad export cookie 2355426957947899855 [ 5781.741323] LustreError: 189238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5781.747646] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5782.062508] Lustre: server umount lustre-MDT0001 complete [ 5790.925498] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:40:23 (1789443623) [ 5798.023388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5798.783742] LustreError: 193171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.895283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.967935] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5804.002042] LustreError: 193172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.121750] LustreError: 193171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.049814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.410749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5815.820103] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5823.355583] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5829.341507] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5836.020951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5836.210794] LustreError: 194993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5836.375146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5841.887689] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5843.453167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5849.390733] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5856.680795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5856.964593] Lustre: lustre-OST0001: new disk, initializing [ 5856.968930] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5856.975686] Lustre: 196011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5857.132352] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5863.670528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5863.698186] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5863.884310] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5864.053410] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5873.582211] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5879.270762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5879.273341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5879.299140] Lustre: Skipped 1 previous similar message [ 5879.320493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5879.338567] Lustre: Skipped 1 previous similar message [ 5883.542960] Lustre: server umount lustre-OST0000 complete [ 5884.393611] LustreError: 194995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.413051] LustreError: 194995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5890.328827] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5890.515496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5894.607177] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5894.843318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5909.989025] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5909.991949] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5909.998332] LustreError: Skipped 1 previous similar message [ 5910.000859] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5910.011767] Lustre: Skipped 2 previous similar messages [ 5910.028064] Lustre: Skipped 2 previous similar messages [ 5913.879320] Lustre: server umount lustre-OST0001 complete [ 5919.212487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5919.227149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.290910] Lustre: server umount lustre-MDT0000 complete [ 5925.351745] LustreError: 193167:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5925.394146] LustreError: 193167:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [ 5928.830640] LustreError: 193153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789443761 with bad export cookie 2355426957947900926 [ 5928.832581] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5928.841867] LustreError: 193153:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5929.193684] Lustre: server umount lustre-MDT0001 complete [ 5953.538538] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:43:05 (1789443785) [ 5959.573697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5959.805540] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5959.838159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5960.023264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5964.239933] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5972.433198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.499609] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5972.546514] Lustre: 199451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5976.552408] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 5983.869859] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5989.689270] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5996.388218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.600419] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6002.596155] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6005.390175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6009.153790] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6015.074348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6015.134611] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6020.062847] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6024.854802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6026.545628] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6030.309071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6030.314339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6030.321647] Lustre: Skipped 2 previous similar messages [ 6030.330246] Lustre: Skipped 4 previous similar messages [ 6035.124364] Lustre: server umount lustre-OST0000 complete [ 6035.429529] LustreError: 201302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.438671] LustreError: 201302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6050.804878] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6056.117752] Lustre: server umount lustre-OST0001 complete [ 6065.933765] Lustre: server umount lustre-MDT0000 complete [ 6068.829347] LustreError: 199444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789443901 with bad export cookie 2355426957947902333 [ 6068.840731] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6068.845838] LustreError: 199444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6069.161332] Lustre: server umount lustre-MDT0001 complete [ 6089.068963] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:45:21 (1789443921) [ 6101.953378] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 6110.727482] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 6147.732743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6147.958118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6147.988420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6148.108639] Lustre: lustre-MDT0000: new disk, initializing [ 6148.178120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6148.182946] Lustre: Skipped 3 previous similar messages [ 6148.204119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6152.097388] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6163.381438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.499674] Lustre: 207548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6163.569382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6163.576045] Lustre: Skipped 1 previous similar message [ 6163.703530] Lustre: lustre-MDT0001: new disk, initializing [ 6163.875061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6163.896350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6168.717931] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6179.214582] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6184.599411] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6190.995844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6191.226110] Lustre: lustre-OST0000: new disk, initializing [ 6191.232435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6191.247522] Lustre: 209517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6197.491150] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6201.403956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6201.414134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6201.500476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6206.170323] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6210.926912] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6211.198719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6215.614167] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6215.830332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6222.310101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6222.311566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6222.326549] Lustre: Skipped 4 previous similar messages [ 6222.333893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6222.336552] LustreError: Skipped 1 previous similar message [ 6222.352712] Lustre: Skipped 9 previous similar messages [ 6223.866377] Lustre: server umount lustre-OST0000 complete [ 6233.593815] Lustre: server umount lustre-MDT0000 complete [ 6236.721889] LustreError: 209516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444069 with bad export cookie 2355426957947904118 [ 6236.725830] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6236.732211] LustreError: 209516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6237.199941] Lustre: server umount lustre-MDT0001 complete [ 6245.381557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.738186] LustreError: 211733:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6245.753866] LustreError: 211733:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 6250.100942] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6258.282201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6262.570849] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6269.980307] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6275.279686] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6281.895891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6282.223475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6282.228153] Lustre: Skipped 4 previous similar messages [ 6289.139325] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6296.561169] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6300.699331] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6300.918450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6305.603980] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6305.882209] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6308.835392] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6308.842487] LustreError: Skipped 1 previous similar message [ 6308.848154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.864587] Lustre: Skipped 3 previous similar messages [ 6308.874949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6308.878954] Lustre: Skipped 4 previous similar messages [ 6313.839727] Lustre: server umount lustre-OST0000 complete [ 6319.900928] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6324.265340] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6353.033388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6360.691829] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6360.711596] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6360.721942] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6369.178783] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6371.908481] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6376.148237] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6376.167582] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6376.419558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6376.433506] LustreError: Skipped 1 previous similar message [ 6382.775470] Lustre: server umount lustre-OST0000 complete [ 6392.678649] Lustre: server umount lustre-MDT0000 complete [ 6396.324753] LustreError: 211713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444229 with bad export cookie 2355426957947905000 [ 6396.327934] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6396.330507] LustreError: 211713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6396.635293] Lustre: server umount lustre-MDT0001 complete [ 6405.524508] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:50:37 (1789444237) [ 6411.160706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6416.410360] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6424.342189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6428.568060] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6436.273455] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6442.301506] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6448.212992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6454.766499] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6461.891602] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6466.182287] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6466.400918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6470.653955] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6470.867765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6476.791667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.957245] Lustre: lustre-OST0001: new disk, initializing [ 6476.963128] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6476.970627] Lustre: 221071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6482.553173] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6485.106258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6485.118672] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6485.189058] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6490.929481] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6495.903905] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6496.144923] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6501.103078] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6501.294326] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6505.963444] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6505.964196] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6505.970597] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6505.970605] Lustre: Skipped 9 previous similar messages [ 6505.976479] Lustre: Skipped 6 previous similar messages [ 6506.009898] LustreError: Skipped 3 previous similar messages [ 6517.216171] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6517.393190] Lustre: server umount lustre-OST0001 complete [ 6521.322427] LustreError: 219998:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6521.342506] LustreError: 219998:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 6524.371376] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6524.584147] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6529.855453] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6530.146208] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6559.229191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6559.515129] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6559.521030] Lustre: Skipped 5 previous similar messages [ 6564.857329] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6564.873054] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6564.876039] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6564.886763] Lustre: Skipped 1 previous similar message [ 6566.193612] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6574.310098] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6576.363170] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6584.902420] Lustre: server umount lustre-OST0001 complete [ 6608.107821] Lustre: server umount lustre-OST0000 complete [ 6618.447537] Lustre: server umount lustre-MDT0000 complete [ 6622.914080] LustreError: 221070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444455 with bad export cookie 2355426957947905966 [ 6622.920268] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6622.933301] LustreError: 221070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6629.323769] Lustre: server umount lustre-MDT0001 complete [ 6655.108699] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:54:47 (1789444487) [ 6663.167683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.541435] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6663.567441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6669.150743] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6678.588729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6678.705192] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6678.751222] Lustre: 226312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6683.596089] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6692.320933] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6699.366248] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6708.015175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.394153] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6714.820880] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6722.310100] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6732.363944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6732.434670] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6739.808820] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6748.662634] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6750.777793] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6755.941352] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6756.153426] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6760.669884] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6760.907542] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6766.856646] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e0987a51800.ost_server_uuid 50 [ 6768.319516] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e0987a51800.ost_server_uuid in IDLE state after 0 sec [ 6769.254753] Lustre: Permanently deactivating lustre-OST0001 [ 6769.271521] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6772.605544] Lustre: Permanently reactivating lustre-OST0001 [ 6777.324740] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6777.327753] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.330866] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6777.330875] Lustre: Skipped 13 previous similar messages [ 6777.346494] LustreError: Skipped 4 previous similar messages [ 6777.365612] Lustre: Skipped 7 previous similar messages [ 6782.209073] Lustre: server umount lustre-OST0001 complete [ 6792.775806] Lustre: server umount lustre-OST0000 complete [ 6802.524749] Lustre: server umount lustre-MDT0000 complete [ 6807.257103] LustreError: 226303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444640 with bad export cookie 2355426957947907247 [ 6807.258639] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6807.271814] LustreError: 226303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6807.524597] Lustre: server umount lustre-MDT0001 complete [ 6835.841260] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:57:48 (1789444668) [ 6844.064415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6844.464939] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6844.502354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6844.516688] Lustre: Skipped 5 previous similar messages [ 6849.326652] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6857.710201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6857.776353] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6857.806066] Lustre: 233009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6862.614685] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6871.105937] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6876.927937] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6884.674455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6884.915696] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6884.939741] Lustre: Permanently deactivating lustre-OST0000 [ 6884.956824] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6892.286610] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6894.618840] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6900.715936] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6908.918858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6908.982566] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6914.573111] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 6918.979724] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6922.563797] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6924.409341] Lustre: Permanently reactivating lustre-OST0000 [ 6924.424150] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6924.434154] Lustre: Skipped 2 previous similar messages [ 6926.172451] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6926.185540] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 00000000fda12604, cur 1789444759 deadline 1789444757 last 1789444727 [ 6926.256182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6940.619360] Lustre: server umount lustre-OST0001 complete [ 6951.498672] Lustre: server umount lustre-OST0000 complete [ 6969.312304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6969.605110] Lustre: server umount lustre-MDT0000 complete [ 6973.394720] LustreError: 233000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789444806 with bad export cookie 2355426957947908920 [ 6973.397329] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6973.402947] LustreError: 233000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6973.751700] Lustre: server umount lustre-MDT0001 complete [ 6985.505930] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 6988.233679] Key type lgssc unregistered [ 6988.541862] LNet: 237891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6988.547930] LNetError: 237891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6989.607575] LNet: Removed LNI 192.168.206.115@tcp [ 6990.514800] Key type .llcrypt unregistered [ 6990.517692] Key type ._llcrypt unregistered [ 7013.874834] Key type ._llcrypt registered [ 7013.879340] Key type .llcrypt registered [ 7013.960685] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7014.579420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7014.627475] alg: No test for adler32 (adler32-zlib) [ 7015.605650] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 7015.860694] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7017.560496] Key type lgssc registered [ 7018.589371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7030.165843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7030.183362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.620220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7036.853703] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7042.024542] LustreError: 239371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7042.037917] LustreError: 239371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7046.552899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7046.684868] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7046.716482] Lustre: 239359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7046.724631] Lustre: Permanently deactivating lustre-MDT0001 [ 7046.738683] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7047.065952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7048.665545] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7048.674145] LustreError: 239401:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7048.692608] LustreError: 239401:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7048.704833] LustreError: 239401:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7051.714561] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7059.119748] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7065.423291] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7072.328815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7072.720712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7076.839385] LustreError: 241212:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7076.856370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7079.297409] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7081.963372] LustreError: 241212:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7087.083482] LustreError: 241212:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7087.186725] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7091.187183] LustreError: 241212:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7091.210398] LustreError: 241212:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7091.213614] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7093.930057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7094.130752] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7099.385873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7099.406239] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7100.589323] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7108.164798] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7112.649758] Lustre: Permanently reactivating lustre-MDT0001 [ 7112.671898] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7112.681902] Lustre: Skipped 2 previous similar messages [ 7117.246896] Lustre: lustre-MDT0001: Client 8ad489ff-ea6b-4e24-b0ae-2007bfa89cd6 (at 192.168.206.15@tcp) reconnecting [ 7123.802584] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7125.769621] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7131.982563] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7132.212887] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7136.647948] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7136.856655] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7139.557849] Lustre: Permanently deactivating lustre-MDT0001 [ 7139.586500] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7139.591244] Lustre: Skipped 2 previous similar messages [ 7146.160321] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7159.781276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7159.788251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7159.807508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7160.802231] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7160.808615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7160.833229] Lustre: Skipped 1 previous similar message [ 7160.851585] Lustre: Skipped 3 previous similar messages [ 7163.677469] Lustre: server umount lustre-MDT0000 complete [ 7165.924367] LustreError: 239370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7165.947476] LustreError: 239370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7167.704769] LustreError: 239349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445000 with bad export cookie 4975521080205783655 [ 7167.705880] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7167.716023] LustreError: 239349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7167.974793] Lustre: server umount lustre-MDT0001 complete [ 7177.696825] Lustre: server umount lustre-OST0000 complete [ 7190.036915] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 00:03:42 (1789445022) [ 7201.676617] Lustre: server umount lustre-OST0001 complete [ 7208.272388] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 7219.463718] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7255.714179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7256.006325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7256.024840] Lustre: Skipped 2 previous similar messages [ 7256.106459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7256.215852] Lustre: lustre-MDT0000: new disk, initializing [ 7256.314825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.330838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7261.284391] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7270.945964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7271.072189] Lustre: 247649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7271.116787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7271.121811] Lustre: Skipped 1 previous similar message [ 7271.211938] Lustre: lustre-MDT0001: new disk, initializing [ 7271.317353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7271.355982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7271.368954] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7275.613470] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7287.080486] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7293.088513] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7300.907946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7301.169984] Lustre: lustre-OST0000: new disk, initializing [ 7301.173710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7301.191427] Lustre: 249614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7301.330519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7308.387150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7308.403043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7308.447581] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7308.539521] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7318.319803] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7320.627378] LustreError: 247654:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7329.541308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7329.752703] Lustre: lustre-OST0001: new disk, initializing [ 7329.756333] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7329.764285] Lustre: 250730:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7330.656097] LustreError: 247654:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7330.689136] LustreError: 247655:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7336.577141] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7339.129238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7339.148640] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7339.258220] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7340.704196] LustreError: 247655:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7347.460853] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7349.538824] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7354.866746] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7354.867640] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7354.871145] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7354.891108] Lustre: Skipped 2 previous similar messages [ 7354.898749] LustreError: Skipped 1 previous similar message [ 7358.293409] Lustre: server umount lustre-OST0001 complete [ 7359.974670] LustreError: 249598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7360.020892] LustreError: 249598:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7380.451553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7380.466602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7380.467915] Lustre: Skipped 1 previous similar message [ 7380.480631] Lustre: Skipped 2 previous similar messages [ 7381.789458] Lustre: server umount lustre-OST0000 complete [ 7385.570963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.581992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7385.596241] Lustre: Skipped 1 previous similar message [ 7391.822078] Lustre: server umount lustre-MDT0000 complete [ 7395.356261] LustreError: 247639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445228 with bad export cookie 4975521080205786476 [ 7395.367880] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7395.370748] LustreError: 247639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7401.685811] Lustre: server umount lustre-MDT0001 complete [ 7414.663098] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7417.629857] Key type lgssc unregistered [ 7417.997352] LNet: 252765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7418.026481] LNetError: 252765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7418.055542] LNet: Removed LNI 192.168.206.115@tcp [ 7418.994097] Key type .llcrypt unregistered [ 7418.998437] Key type ._llcrypt unregistered [ 7461.229136] Key type ._llcrypt registered [ 7461.231250] Key type .llcrypt registered [ 7461.346528] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7462.953793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7462.981151] alg: No test for adler32 (adler32-zlib) [ 7464.097946] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 7464.320431] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7465.992272] Key type lgssc registered [ 7467.378243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7476.258916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7476.286777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.633942] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7477.660367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7477.931145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.600761] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7491.832423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7491.951064] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7491.995625] Lustre: 254971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7492.170966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7496.498065] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7503.532737] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7510.045914] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7517.493286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7517.622320] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7517.769056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.344737] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7528.058241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7531.213561] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7584.744067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7584.746515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7584.762688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7589.345802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7589.354979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.370969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7590.803559] Lustre: server umount lustre-OST0000 complete [ 7595.295965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7609.388772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7609.742138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7614.903915] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7614.931341] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7616.984915] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7624.828399] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7629.289885] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7629.322569] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7665.125163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7665.141327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.153510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7665.156984] Lustre: Skipped 1 previous similar message [ 7668.195741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7668.199255] Lustre: Skipped 1 previous similar message [ 7670.946248] Lustre: server umount lustre-OST0000 complete [ 7676.385397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7676.393907] LustreError: Skipped 1 previous similar message [ 7676.404268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.415981] Lustre: Skipped 1 previous similar message [ 7676.420352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7681.091600] Lustre: server umount lustre-MDT0000 complete [ 7685.144303] LustreError: 258581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445517 with bad export cookie 887305424261559215 [ 7685.148737] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7685.156610] LustreError: 258581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7685.567679] Lustre: server umount lustre-MDT0001 complete [ 7695.945840] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7698.568895] Key type lgssc unregistered [ 7698.841226] LNet: 260391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7698.849571] LNetError: 260391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7698.869147] LNet: Removed LNI 192.168.206.115@tcp [ 7699.788712] Key type .llcrypt unregistered [ 7699.791488] Key type ._llcrypt unregistered [ 7722.491714] Key type ._llcrypt registered [ 7722.494605] Key type .llcrypt registered [ 7722.597269] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7723.412803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7723.428430] alg: No test for adler32 (adler32-zlib) [ 7724.366731] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 7724.628316] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7726.336876] Key type lgssc registered [ 7727.685825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7736.229979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7736.262295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7737.946876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7742.706402] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7748.077171] LustreError: 261805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7748.112365] LustreError: 261805:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7753.049175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.190838] LustreError: 261804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7753.358634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7758.249641] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7765.109757] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7770.851360] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7776.832909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7777.110255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7782.254461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7782.629702] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7790.656933] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7818.209560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7818.220500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7818.223323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7818.242232] Lustre: Skipped 1 previous similar message [ 7818.254293] Lustre: Skipped 1 previous similar message [ 7822.856691] Lustre: server umount lustre-OST0000 complete [ 7828.458566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7828.465228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7828.473124] Lustre: Skipped 1 previous similar message [ 7832.406264] Lustre: server umount lustre-MDT0000 complete [ 7833.570054] LustreError: 261805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7833.591641] LustreError: 261805:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7836.100173] LustreError: 261786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445668 with bad export cookie 10441352596185987741 [ 7836.106405] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7836.117963] LustreError: 261786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7836.507479] Lustre: server umount lustre-MDT0001 complete [ 7848.967937] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7852.125933] Key type lgssc unregistered [ 7852.530659] LNet: 266111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7852.541605] LNetError: 266111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7852.561112] LNet: Removed LNI 192.168.206.115@tcp [ 7853.543551] Key type .llcrypt unregistered [ 7853.546291] Key type ._llcrypt unregistered [ 7871.801776] Key type ._llcrypt registered [ 7871.806274] Key type .llcrypt registered [ 7871.910834] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 7872.743315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7872.813060] alg: No test for adler32 (adler32-zlib) [ 7873.787757] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 7874.109437] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 7875.792251] Key type lgssc registered [ 7877.005661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7884.813404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7884.827979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7886.143409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7891.113817] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7895.528772] LustreError: 267329:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7895.544313] LustreError: 267329:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7899.840708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7900.234784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7905.112678] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7913.294492] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7918.977808] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7926.003303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7926.451499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7930.565720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7933.615070] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 7941.463142] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7955.428137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7955.434225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7955.448711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7955.468350] Lustre: Skipped 1 previous similar message [ 7960.560608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7960.571232] Lustre: Skipped 1 previous similar message [ 7960.880447] Lustre: server umount lustre-OST0000 complete [ 7965.667795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7965.671189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7965.688241] Lustre: Skipped 2 previous similar messages [ 7970.786825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7970.798516] Lustre: Skipped 2 previous similar messages [ 7971.118453] Lustre: server umount lustre-MDT0000 complete [ 7974.715259] LustreError: 269172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445807 with bad export cookie 9401395608853243112 [ 7974.721340] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7974.722659] LustreError: 269172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7975.020700] Lustre: server umount lustre-MDT0001 complete [ 7985.641950] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 7988.035508] Key type lgssc unregistered [ 7988.310617] LNet: 271006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7988.324378] LNetError: 271006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7988.345775] LNet: Removed LNI 192.168.206.115@tcp [ 7989.565216] Key type .llcrypt unregistered [ 7989.571387] Key type ._llcrypt unregistered [ 8014.325907] Key type ._llcrypt registered [ 8014.327749] Key type .llcrypt registered [ 8014.402874] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8015.264739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8015.388922] alg: No test for adler32 (adler32-zlib) [ 8016.554369] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 8016.808035] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8018.473211] Key type lgssc registered [ 8019.202789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8026.069517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8026.096295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8027.520578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8031.494425] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8037.865754] LustreError: 272419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8037.906131] LustreError: 272419:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8040.937514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8041.637425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8046.876823] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8055.722075] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8061.897691] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8069.255663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8069.570575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8072.637734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8076.718843] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8086.037973] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8112.610941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8112.616841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8112.618989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8112.624872] Lustre: Skipped 1 previous similar message [ 8112.636098] Lustre: Skipped 1 previous similar message [ 8115.570548] Lustre: server umount lustre-OST0000 complete [ 8122.853593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.859980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8122.865536] Lustre: Skipped 1 previous similar message [ 8122.874921] Lustre: Skipped 1 previous similar message [ 8125.357942] LustreError: 275975:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (000000004845dfaf) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 8125.427189] Lustre: server umount lustre-MDT0000 complete [ 8127.969686] LustreError: 273595:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8127.987536] LustreError: 273595:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8128.441424] LustreError: 272400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789445961 with bad export cookie 4305478481216926877 [ 8128.453708] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8128.670397] Lustre: server umount lustre-MDT0001 complete [ 8137.844295] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8140.252272] Key type lgssc unregistered [ 8140.566993] LNet: 276725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8140.579867] LNetError: 276725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8141.607658] LNet: Removed LNI 192.168.206.115@tcp [ 8142.307506] Key type .llcrypt unregistered [ 8142.310089] Key type ._llcrypt unregistered [ 8161.610553] Key type ._llcrypt registered [ 8161.612272] Key type .llcrypt registered [ 8161.780200] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8163.236575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8163.342045] alg: No test for adler32 (adler32-zlib) [ 8164.480363] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 8164.733879] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8166.392244] Key type lgssc registered [ 8167.152730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8173.466636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8173.483516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8174.867548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8178.859668] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8184.302312] LustreError: 277957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8184.323609] LustreError: 277957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8186.886424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8187.414078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8190.833676] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8198.192659] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8203.876602] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8210.436121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8210.805875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8216.717218] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8220.035548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8224.527171] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8235.489557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8235.498255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8235.508813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8238.570721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8238.573430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.576941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8241.781060] Lustre: server umount lustre-OST0000 complete [ 8248.811193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8248.815842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.820995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8248.821027] Lustre: Skipped 1 previous similar message [ 8248.873482] Lustre: Skipped 1 previous similar message [ 8251.731249] Lustre: server umount lustre-MDT0000 complete [ 8253.922647] LustreError: 277964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.954424] LustreError: 277964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8255.293394] LustreError: 277922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446088 with bad export cookie 13917103580332401215 [ 8255.307523] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8255.309594] LustreError: 277922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8255.670629] Lustre: server umount lustre-MDT0001 complete [ 8267.532868] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8270.269973] Key type lgssc unregistered [ 8270.527832] LNet: 281655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8270.534297] LNetError: 281655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8270.558188] LNet: Removed LNI 192.168.206.115@tcp [ 8271.353692] Key type .llcrypt unregistered [ 8271.356516] Key type ._llcrypt unregistered [ 8282.876664] Key type ._llcrypt registered [ 8282.883816] Key type .llcrypt registered [ 8283.028734] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8300.004772] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 8315.695419] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8316.490912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8316.552245] alg: No test for adler32 (adler32-zlib) [ 8317.687801] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 8317.952647] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8319.664330] Key type lgssc registered [ 8321.113412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8354.333350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8354.353299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8355.761910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8355.825875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8355.972688] Lustre: lustre-MDT0000: new disk, initializing [ 8356.079901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8356.097861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8361.249427] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8373.971466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8374.145164] Lustre: 285203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8374.246580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8374.258038] Lustre: Skipped 1 previous similar message [ 8374.376294] Lustre: lustre-MDT0001: new disk, initializing [ 8374.540658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8374.618605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8374.646848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8379.193661] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8388.477908] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8394.592520] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8401.979250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8402.267647] Lustre: lustre-OST0000: new disk, initializing [ 8402.273452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8402.286503] Lustre: 287174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8402.374653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8409.252936] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8410.217260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8410.252676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8410.427473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8420.082462] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8425.452759] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8425.784984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8431.451238] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8431.730994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8436.198313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8436.205636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8436.226199] LustreError: Skipped 1 previous similar message [ 8436.242554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8436.248623] Lustre: Skipped 1 previous similar message [ 8440.347213] Lustre: server umount lustre-OST0000 complete [ 8446.439103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8446.445832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8446.447838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8446.466491] Lustre: Skipped 2 previous similar messages [ 8450.567644] Lustre: server umount lustre-MDT0000 complete [ 8454.601366] LustreError: 285195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446287 with bad export cookie 3209761597477310767 [ 8454.605527] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8454.618281] LustreError: 285195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8455.028809] Lustre: server umount lustre-MDT0001 complete [ 8464.975891] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 00:24:57 (1789446297) [ 8471.891344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8472.319919] LustreError: 289590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8472.349959] LustreError: 289590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8472.421797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8476.800895] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8477.665222] LustreError: 289591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8482.788104] LustreError: 289590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8486.334517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.863506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8492.130165] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8500.884274] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8507.985135] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8515.597468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8515.894978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.264536] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8530.329978] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8539.618157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8539.646416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.688530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8539.693678] Lustre: Skipped 1 previous similar message [ 8541.667400] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8541.676548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8545.665756] Lustre: server umount lustre-OST0000 complete [ 8551.907747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8551.911490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8551.938339] Lustre: Skipped 1 previous similar message [ 8551.943182] Lustre: Skipped 2 previous similar messages [ 8555.957763] Lustre: server umount lustre-MDT0000 complete [ 8557.028979] LustreError: 289586:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8557.048320] LustreError: 289586:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8559.585816] LustreError: 289570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446392 with bad export cookie 3209761597477311649 [ 8559.591236] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8559.599769] LustreError: 289570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8559.997438] Lustre: server umount lustre-MDT0001 complete [ 8570.112847] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 8572.492460] Key type lgssc unregistered [ 8572.783774] LNet: 293123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8572.799376] LNetError: 293123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8573.863433] LNet: Removed LNI 192.168.206.115@tcp [ 8574.703563] Key type .llcrypt unregistered [ 8574.706209] Key type ._llcrypt unregistered [ 8600.111764] Key type ._llcrypt registered [ 8600.114524] Key type .llcrypt registered [ 8600.306436] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 8615.279950] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8616.482164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8616.575054] alg: No test for adler32 (adler32-zlib) [ 8617.562036] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 8617.827080] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 8619.536518] Key type lgssc registered [ 8620.466775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8667.916596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8667.963564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8669.140147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8669.169656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8669.228478] Lustre: lustre-MDT0000: new disk, initializing [ 8669.285893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8669.326659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8672.672928] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8682.210759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8682.314558] Lustre: 296948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8682.367442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8682.370167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8682.372616] Lustre: Skipped 1 previous similar message [ 8682.431740] Lustre: lustre-MDT0001: new disk, initializing [ 8682.504173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8682.537219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8682.546757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8686.865966] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8695.373945] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8701.140488] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8708.498821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8708.709838] Lustre: lustre-OST2710: new disk, initializing [ 8708.713441] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8708.719909] Lustre: 298916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8708.789545] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8714.303378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8714.312527] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8714.454340] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8715.211847] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8723.677971] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8729.340380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8729.505099] Lustre: lustre-OST03e8: new disk, initializing [ 8729.508537] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8729.514305] Lustre: 299982:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8729.608491] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8735.749420] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8738.386070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8738.396694] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8738.476100] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8742.231647] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8749.417710] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8749.609626] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8753.414340] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8753.608877] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8758.847853] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8759.092097] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8763.339927] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8763.622853] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8769.515249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8769.517956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8769.520140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8769.538625] Lustre: Skipped 2 previous similar messages [ 8769.559975] Lustre: Skipped 3 previous similar messages [ 8773.528788] Lustre: server umount lustre-MDT0000 complete [ 8776.519920] LustreError: 296942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446609 with bad export cookie 8686032476092503669 [ 8776.529197] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8776.530486] LustreError: 296942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8776.925041] Lustre: server umount lustre-MDT0001 complete [ 8790.972224] Lustre: server umount lustre-OST2710 complete [ 8805.783925] Lustre: server umount lustre-OST03e8 complete [ 8825.652934] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 8835.343671] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 8871.785136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.002237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8872.036699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8872.116712] Lustre: lustre-MDT0000: new disk, initializing [ 8872.194241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8872.206471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8876.330902] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8886.468105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8886.596876] Lustre: 305230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8886.639725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8886.643243] Lustre: Skipped 1 previous similar message [ 8886.710273] Lustre: lustre-MDT0001: new disk, initializing [ 8886.789307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8886.830752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8886.842701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8891.003585] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8899.724100] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8905.380148] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8912.238708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8912.529574] Lustre: lustre-OST0000: new disk, initializing [ 8912.536423] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8912.545271] Lustre: 307200:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8912.621562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8918.838979] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 8921.156444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8921.193412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8921.300975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8928.392481] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8933.102902] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8933.338764] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8937.893682] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8938.140593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8942.049177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8942.056925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8942.065627] LustreError: Skipped 1 previous similar message [ 8942.074607] Lustre: Skipped 1 previous similar message [ 8942.080024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8942.107700] Lustre: Skipped 1 previous similar message [ 8947.171088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8947.183832] Lustre: Skipped 1 previous similar message [ 8952.324967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8952.329631] Lustre: Skipped 1 previous similar message [ 8954.336129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8954.548995] Lustre: server umount lustre-OST0000 complete [ 8958.439955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8958.446325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8958.462229] Lustre: Skipped 1 previous similar message [ 8962.532558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8962.534056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8962.545093] Lustre: Skipped 1 previous similar message [ 8964.526575] Lustre: server umount lustre-MDT0000 complete [ 8967.651848] LustreError: 305238:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8967.681497] LustreError: 305238:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 8968.216686] LustreError: 307199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446801 with bad export cookie 8686032476092505349 [ 8968.227304] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8968.229709] LustreError: 307199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8968.579280] Lustre: server umount lustre-MDT0001 complete [ 8976.505636] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 00:33:28 (1789446808) [ 8977.933895] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8979.558607] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 00:33:32 (1789446812) [ 8982.828417] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9005.697848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9005.944378] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9005.968781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9006.120515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9011.068289] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9019.443590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9019.527498] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9019.556245] Lustre: 311057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9023.296538] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9042.478314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9042.701424] Lustre: Denying initial registration attempt from nid 192.168.206.115@tcp, specified as failover [ 9042.709669] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9042.718755] LustreError: 312510:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9042.727865] LustreError: 312510:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9042.736764] LustreError: 312510:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9042.815979] Lustre: server umount lustre-OST0000 complete [ 9042.822750] LustreError: 312510:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9048.548333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9048.563929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9048.573943] Lustre: Skipped 2 previous similar messages [ 9054.734233] Lustre: server umount lustre-MDT0000 complete [ 9057.861134] LustreError: 311051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789446890 with bad export cookie 8686032476092506231 [ 9057.867352] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9057.873156] LustreError: 311051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9058.281654] Lustre: server umount lustre-MDT0001 complete [ 9068.265730] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9070.555400] Key type lgssc unregistered [ 9070.826865] LNet: 313631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9070.837683] LNetError: 313631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9070.858554] LNet: Removed LNI 192.168.206.115@tcp [ 9071.569055] Key type .llcrypt unregistered [ 9071.570740] Key type ._llcrypt unregistered [ 9082.975143] Key type ._llcrypt registered [ 9082.976824] Key type .llcrypt registered [ 9083.080101] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9083.814349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9083.951644] alg: No test for adler32 (adler32-zlib) [ 9085.054467] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 9085.301609] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9086.952216] Key type lgssc registered [ 9087.999340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9133.098379] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9141.729123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9141.754610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9143.062356] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9143.114926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9143.280354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9147.723544] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9155.422581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9155.591977] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9155.621654] Lustre: 316396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9155.826802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9159.504576] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9166.640460] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9172.004589] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9191.658290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.865250] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9192.056021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9197.494167] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9204.378530] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9209.827324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9209.840578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9209.868486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9210.350140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9210.374864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9210.398114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9210.403210] Lustre: Skipped 1 previous similar message [ 9213.205182] Lustre: server umount lustre-OST0000 complete [ 9217.505977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9217.527126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9217.549264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.587840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.588918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9223.764868] Lustre: server umount lustre-MDT0000 complete [ 9227.671324] LustreError: 316386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447060 with bad export cookie 12583885218785539748 [ 9227.682344] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9227.689845] LustreError: 316386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9228.073543] Lustre: server umount lustre-MDT0001 complete [ 9239.707107] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9242.783534] Key type lgssc unregistered [ 9243.085731] LNet: 319967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9243.094091] LNetError: 319967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9243.108796] LNet: Removed LNI 192.168.206.115@tcp [ 9243.889325] Key type .llcrypt unregistered [ 9243.894125] Key type ._llcrypt unregistered [ 9268.402428] Key type ._llcrypt registered [ 9268.405875] Key type .llcrypt registered [ 9268.532755] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9269.354514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9269.394274] alg: No test for adler32 (adler32-zlib) [ 9270.505775] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 9270.735414] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9272.400266] Key type lgssc registered [ 9273.250822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9280.660859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9280.699168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9282.086757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9286.586631] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9291.239990] LustreError: 321383:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9291.254353] LustreError: 321383:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9294.581794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9294.995475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9299.539858] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9307.867855] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9313.241640] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9321.898342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9322.394325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9328.017258] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9335.811954] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9350.117710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9350.137721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9350.142666] Lustre: Skipped 2 previous similar messages [ 9350.147498] Lustre: Skipped 1 previous similar message [ 9354.164692] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [ 9355.238672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9355.244672] Lustre: Skipped 2 previous similar messages [ 9356.212358] Lustre: server umount lustre-MDT0000 complete [ 9359.270243] LustreError: 321378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9359.622058] LustreError: 321362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447192 with bad export cookie 3910938705383492359 [ 9359.625532] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9359.632580] LustreError: 321362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9360.353444] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9360.355441] LustreError: 321377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9360.369771] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9364.391949] LustreError: 321379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9364.393674] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9364.407259] LustreError: 321379:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9366.071824] Lustre: server umount lustre-MDT0001 complete [ 9372.198672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9380.779664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9384.934477] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3646720810b5a7f5 [ 9384.956752] Lustre: MGC192.168.206.115@tcp: Connection restored to 0@lo (at 0@lo) [ 9385.265479] LustreError: 323213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9385.331798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9385.428407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9390.078254] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9394.532612] LustreError: 324997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9394.545733] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9394.564492] LustreError: 324997:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9398.615663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9399.035565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9403.577932] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9404.391403] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9410.524202] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9411.906082] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9417.397996] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9418.742569] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9424.869464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9424.871966] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9424.882588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9424.882595] Lustre: Skipped 1 previous similar message [ 9424.911408] Lustre: Skipped 1 previous similar message [ 9429.481995] Lustre: server umount lustre-OST0000 complete [ 9435.105780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9435.128167] Lustre: Skipped 1 previous similar message [ 9439.738641] Lustre: server umount lustre-MDT0000 complete [ 9440.227860] LustreError: 325002:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9440.249142] LustreError: 325002:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9443.318055] LustreError: 321363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447276 with bad export cookie 3910938705383499765 [ 9443.322993] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9443.328756] LustreError: 321363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9443.627776] Lustre: server umount lustre-MDT0001 complete [ 9454.269599] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9456.779774] Key type lgssc unregistered [ 9457.126446] LNet: 327670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9457.141975] LNetError: 327670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9457.154326] LNet: Removed LNI 192.168.206.115@tcp [ 9457.900187] Key type .llcrypt unregistered [ 9457.902167] Key type ._llcrypt unregistered [ 9480.131924] Key type ._llcrypt registered [ 9480.137622] Key type .llcrypt registered [ 9480.259521] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9481.184720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9481.291660] alg: No test for adler32 (adler32-zlib) [ 9482.402823] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 9482.635429] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9484.336222] Key type lgssc registered [ 9485.327496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9493.233243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9493.245775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9494.766039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9499.131519] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9504.227338] LustreError: 329085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9504.246520] LustreError: 329085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9508.542240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9509.291624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9514.558870] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9522.758450] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9529.400950] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9533.751457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9533.769067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9533.778195] Lustre: Skipped 1 previous similar message [ 9533.794480] Lustre: Skipped 1 previous similar message [ 9539.041681] LustreError: 329079:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9539.060182] LustreError: 329079:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9539.124549] Lustre: server umount lustre-MDT0000 complete [ 9542.608989] LustreError: 329064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447375 with bad export cookie 5236390812996360081 [ 9542.613284] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9542.622981] LustreError: 329064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9542.997921] Lustre: server umount lustre-MDT0001 complete [ 9556.735858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9557.009570] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9557.041953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9557.296305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9561.573461] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9570.294436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9570.428629] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9570.491546] Lustre: 331673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9570.812200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9575.224116] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9582.617785] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9589.145344] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9599.030181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9599.200562] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9599.213269] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9599.220864] LustreError: 333559:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9599.226448] LustreError: 333559:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9599.233051] LustreError: 333559:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9599.296567] Lustre: server umount lustre-OST0000 complete [ 9599.305684] LustreError: 333559:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9609.587111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9609.822794] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9610.056036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9615.502180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9616.687584] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9625.212629] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9634.106404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9634.232737] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9634.305880] Lustre: lustre-OST0001: new disk, initializing [ 9634.308757] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9634.313776] Lustre: 335097:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9634.400766] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9640.664523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9640.680679] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9640.817463] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9641.323950] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9648.795967] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9656.297619] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9656.301202] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9656.303905] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9656.320167] Lustre: Skipped 1 previous similar message [ 9658.057056] Lustre: server umount lustre-OST0001 complete [ 9661.430765] LustreError: 334024:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9661.459578] LustreError: 334024:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9666.531175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9666.532100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9666.558391] Lustre: Skipped 1 previous similar message [ 9666.565636] LustreError: 334025:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9666.570592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9666.570610] Lustre: Skipped 1 previous similar message [ 9666.614629] LustreError: 334025:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9668.848269] Lustre: server umount lustre-OST0000 complete [ 9676.771609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9676.790422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9676.793560] Lustre: Skipped 2 previous similar messages [ 9680.278083] Lustre: server umount lustre-MDT0000 complete [ 9681.901764] LustreError: 331680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9681.942429] LustreError: 331680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9684.822637] LustreError: 331665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447517 with bad export cookie 5236390812996360634 [ 9684.834775] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9684.844350] LustreError: 331665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9685.351539] Lustre: server umount lustre-MDT0001 complete [ 9696.800942] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [ 9699.424857] Key type lgssc unregistered [ 9699.757191] LNet: 337056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9699.766989] LNetError: 337056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9699.786069] LNet: Removed LNI 192.168.206.115@tcp [ 9700.766695] Key type .llcrypt unregistered [ 9700.771363] Key type ._llcrypt unregistered [ 9772.957499] Key type ._llcrypt registered [ 9772.961974] Key type .llcrypt registered [ 9773.069115] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [ 9792.040396] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [ 9792.870758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9792.887277] alg: No test for adler32 (adler32-zlib) [ 9793.845606] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [ 9794.014283] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [ 9795.681438] Key type lgssc registered [ 9796.813721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9832.135829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9832.191056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9833.604715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9833.633829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9833.708480] Lustre: lustre-MDT0000: new disk, initializing [ 9833.815137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9833.840546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9839.173512] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9851.305762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.398253] Lustre: 341990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9851.452863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9851.456932] Lustre: Skipped 1 previous similar message [ 9851.528330] Lustre: lustre-MDT0001: new disk, initializing [ 9851.598407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9851.638993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9851.656220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9856.568156] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9870.479892] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9878.455844] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9885.769390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9886.119648] Lustre: lustre-OST0000: new disk, initializing [ 9886.125162] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9886.143482] Lustre: 343959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9886.248669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9892.397506] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9894.965205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9894.977732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9895.081402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9902.297991] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9906.739124] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9906.959948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9911.720073] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9911.979399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9915.881278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9915.892127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9915.895027] LustreError: Skipped 1 previous similar message [ 9915.906563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9915.914786] Lustre: Skipped 1 previous similar message [ 9920.560116] Lustre: server umount lustre-OST0000 complete [ 9926.115124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9926.127368] Lustre: Skipped 1 previous similar message [ 9926.130955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9926.139276] Lustre: Skipped 1 previous similar message [ 9931.035756] Lustre: server umount lustre-MDT0000 complete [ 9934.717774] LustreError: 341981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447767 with bad export cookie 2047862467079218159 [ 9934.719592] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9934.729274] LustreError: 341981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9935.109345] Lustre: server umount lustre-MDT0001 complete [ 9942.636682] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:49:35 (1789447775) [ 9949.824634] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:49:42 (1789447782) [ 9958.590971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.341513] LustreError: 346714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9959.375661] LustreError: 346714:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9959.417208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9964.329325] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9964.519035] LustreError: 346715:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9969.637126] LustreError: 346714:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9973.179859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.406024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9977.552155] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [ 9984.708744] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9990.527173] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9997.079018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9997.436222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10003.674995] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10011.168764] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10017.920416] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10019.613150] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10021.070664] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10022.551590] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10024.705831] Lustre: Failing over lustre-MDT0000 [10025.034944] Lustre: server umount lustre-MDT0000 complete [10025.445588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10025.446853] LustreError: 346711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10025.478157] Lustre: Skipped 3 previous similar messages [10025.505259] LustreError: 346711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10030.562494] LustreError: 346715:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10030.580637] LustreError: 346715:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10040.802049] LustreError: 349304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10040.817706] LustreError: 349304:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10041.827079] Lustre: 340211:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789447858/real 1789447858] req@00000000eb58e5e8 x1876371833951744/t0(0) o400->MGC192.168.206.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789447874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10041.866499] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10044.308765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10052.065680] LustreError: 340207:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003b14af8f x1876371833957888/t0(0) o250->MGC192.168.206.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10052.315689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10052.361663] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10053.535712] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10056.845763] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10057.711193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10057.753544] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10057.792071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10066.372531] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10068.007085] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10071.733759] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10074.140196] Lustre: Failing over lustre-MDT0000 [10074.562845] Lustre: server umount lustre-MDT0000 complete [10078.186251] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10078.209521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10078.211438] LustreError: 346711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10078.268965] LustreError: 346711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [10093.769498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.883593] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10094.212404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10094.305197] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10094.334572] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10098.770524] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10099.700521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10099.719948] Lustre: Skipped 2 previous similar messages [10099.774734] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10099.838227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10108.931324] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10110.608202] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10120.163786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10120.168497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10120.190874] Lustre: Skipped 3 previous similar messages [10120.195755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10120.212433] Lustre: Skipped 2 previous similar messages [10123.737034] Lustre: server umount lustre-MDT0000 complete [10125.284623] LustreError: 346709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10125.303163] LustreError: 346709:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [10127.470344] LustreError: 346696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447960 with bad export cookie 2047862467079220868 [10127.484722] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10127.486022] LustreError: 346696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10127.789845] Lustre: server umount lustre-MDT0001 complete [10141.364609] Lustre: server umount lustre-OST0000 complete [10156.508198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10160.776256] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10169.720334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10170.186224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10170.191430] Lustre: Skipped 1 previous similar message [10174.592340] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10182.779180] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10188.676404] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10196.237119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10201.730661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10203.280824] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10210.740900] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10220.514440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10220.523222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10220.538178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10220.541303] Lustre: Skipped 1 previous similar message [10220.563962] Lustre: Skipped 1 previous similar message [10225.635092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10225.640151] Lustre: Skipped 1 previous similar message [10226.551640] Lustre: server umount lustre-OST0000 complete [10235.878341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10235.890536] Lustre: Skipped 3 previous similar messages [10237.089988] Lustre: server umount lustre-MDT0000 complete [10240.421633] LustreError: 353036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448073 with bad export cookie 2047862467079221610 [10240.445858] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10240.785491] Lustre: server umount lustre-MDT0001 complete [10252.698259] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [10256.697341] Key type lgssc unregistered [10257.150602] LNet: 356588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10257.158890] LNetError: 356588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10257.201223] LNet: Removed LNI 192.168.206.115@tcp [10258.417115] Key type .llcrypt unregistered [10258.429747] Key type ._llcrypt unregistered [10286.266364] Key type ._llcrypt registered [10286.268308] Key type .llcrypt registered [10286.361737] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [10287.370421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10287.421756] alg: No test for adler32 (adler32-zlib) [10288.404059] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [10288.659908] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [10290.352279] Key type lgssc registered [10291.379255] Lustre: Echo OBD driver; http://www.lustre.org/ [10298.854257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10298.869484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.321031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10304.263318] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10310.627264] LustreError: 358101:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10310.640394] LustreError: 358101:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10311.934342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10312.240041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10316.730593] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10323.993883] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10329.335472] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10336.240293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10336.455090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10342.757760] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10345.729349] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10350.610734] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10361.319838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10361.322776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10361.334817] Lustre: Skipped 2 previous similar messages [10361.345263] Lustre: Skipped 2 previous similar messages [10363.566170] Lustre: server umount lustre-MDT0000 complete [10366.434681] LustreError: 360591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10366.470930] LustreError: 360591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10367.711753] LustreError: 358081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448200 with bad export cookie 9265746153107808354 [10367.727076] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10368.036384] Lustre: server umount lustre-MDT0001 complete [10383.112711] Lustre: server umount lustre-OST0000 complete [10398.510706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10399.145236] LustreError: 361813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10399.224116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10403.838547] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10404.323966] LustreError: 361814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10409.452690] LustreError: 361813:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10415.170772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10415.741692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10422.268817] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10431.273896] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10437.797758] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10445.269917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10445.614551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10451.155059] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10458.111078] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10470.206221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10487.266929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10487.273682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10487.285909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10490.850873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10490.863617] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10492.750382] Lustre: server umount lustre-MDT0000 complete [10495.974878] LustreError: 361814:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10495.995880] LustreError: 361814:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10496.816419] LustreError: 361796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448329 with bad export cookie 9265746153107809467 [10496.835264] LustreError: 361796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10496.838318] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10497.174128] Lustre: server umount lustre-MDT0001 complete [10512.125820] Lustre: server umount lustre-OST0000 complete [10529.987674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10530.648703] LustreError: 365749:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.728938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10536.093136] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10546.013776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10551.062491] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10559.057772] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10564.990356] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10571.698036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10572.080115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10572.082721] Lustre: Skipped 1 previous similar message [10576.180278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10579.096349] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10587.200745] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10596.833322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10596.843477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10596.854380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10596.863919] Lustre: Skipped 2 previous similar messages [10596.879544] Lustre: Skipped 3 previous similar messages [10598.246621] Lustre: server umount lustre-OST0000 complete [10602.977632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10602.984226] LustreError: Skipped 1 previous similar message [10602.990746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10603.008799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10608.515809] Lustre: server umount lustre-MDT0000 complete [10612.204796] LustreError: 365750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10612.235265] LustreError: 365750:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10612.589143] LustreError: 365731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448445 with bad export cookie 9265746153107810489 [10612.593808] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10612.610198] LustreError: 365731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10612.934290] Lustre: server umount lustre-MDT0001 complete [10624.436562] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [10627.401846] Key type lgssc unregistered [10627.788103] LNet: 369279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10627.794394] LNetError: 369279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10627.812410] LNet: Removed LNI 192.168.206.115@tcp [10629.021335] Key type .llcrypt unregistered [10629.023241] Key type ._llcrypt unregistered [10655.877535] Key type ._llcrypt registered [10655.878894] Key type .llcrypt registered [10655.983164] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [10656.927063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10656.986108] alg: No test for adler32 (adler32-zlib) [10657.963402] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [10658.134299] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [10659.766800] Key type lgssc registered [10660.785952] Lustre: Echo OBD driver; http://www.lustre.org/ [10667.990186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10668.003851] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.009310] LustreError: 370729:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10668.022360] LustreError: 370729:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10668.027641] LustreError: 370729:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10668.032960] LustreError: 370729:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10668.038183] LustreError: 370729:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10676.120402] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10676.134666] LustreError: 371126:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10676.179037] LustreError: 371126:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10676.195490] LustreError: 371126:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10676.205972] LustreError: 371126:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10676.217504] LustreError: 371126:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10697.237546] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [10700.161820] Key type lgssc unregistered [10700.479912] LNet: 372130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10700.492737] LNetError: 372130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10700.519087] LNet: Removed LNI 192.168.206.115@tcp [10701.372647] Key type .llcrypt unregistered [10701.378037] Key type ._llcrypt unregistered [10721.438261] Key type ._llcrypt registered [10721.442581] Key type .llcrypt registered [10721.531430] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [10740.537241] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [10741.984192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10742.054964] alg: No test for adler32 (adler32-zlib) [10743.166861] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [10743.420317] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [10745.136309] Key type lgssc registered [10746.388698] Lustre: Echo OBD driver; http://www.lustre.org/ [10785.966851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10785.999527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10787.344916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10787.384606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10787.495378] Lustre: lustre-MDT0000: new disk, initializing [10787.630840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10787.652373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10792.462183] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10802.945475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10803.131414] Lustre: 375286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10803.197452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10803.203025] Lustre: Skipped 1 previous similar message [10803.303965] Lustre: lustre-MDT0001: new disk, initializing [10803.442856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10803.498045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10803.517821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10808.539264] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10818.782770] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10825.628507] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10832.882082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10833.130689] Lustre: lustre-OST0000: new disk, initializing [10833.133520] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10833.140679] Lustre: 377253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10833.231047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10839.035851] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10842.681960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10842.688959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10842.876489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10849.178597] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10853.979625] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10854.191467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10858.700654] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10858.949231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10863.590373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10863.595665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10863.597381] LustreError: Skipped 1 previous similar message [10863.610359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10863.625941] Lustre: Skipped 1 previous similar message [10863.652568] Lustre: Skipped 1 previous similar message [10868.710591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10868.716220] Lustre: Skipped 1 previous similar message [10873.833676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10875.360510] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10875.541283] Lustre: server umount lustre-OST0000 complete [10878.946854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10878.947562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10878.957261] Lustre: Skipped 1 previous similar message [10878.965313] Lustre: Skipped 1 previous similar message [10884.068942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10884.085515] Lustre: Skipped 1 previous similar message [10893.280121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10893.747362] Lustre: server umount lustre-MDT0000 complete [10897.677434] LustreError: 375276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448730 with bad export cookie 6447104103721559549 [10897.687622] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10898.053259] Lustre: server umount lustre-MDT0001 complete [10909.752320] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10911.724091] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:05:43 (1789448743) [10919.938458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10920.390689] LustreError: 379716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10920.416080] LustreError: 379716:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10920.480600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10925.459732] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10925.548816] LustreError: 379717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.662350] LustreError: 379716:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.787426] LustreError: 379717:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.946440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10936.336295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10941.153786] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10949.871808] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10956.242180] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10963.541616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10963.816263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10971.328083] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10979.744640] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10988.090852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10988.309279] Lustre: lustre-OST0001: new disk, initializing [10988.324156] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10988.340479] Lustre: 382554:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10988.505722] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10994.969640] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [10995.318896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10995.328952] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10995.429514] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11004.540467] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11011.043124] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11011.048484] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11011.067709] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11011.077645] Lustre: Skipped 3 previous similar messages [11014.714134] Lustre: server umount lustre-OST0001 complete [11016.172814] LustreError: 381540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11016.206182] LustreError: 381540:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11026.405877] LustreError: 383440:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11026.431377] LustreError: 383440:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11032.546350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11032.547535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11032.557184] LustreError: Skipped 1 previous similar message [11032.559496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11032.568907] Lustre: Skipped 2 previous similar messages [11032.580986] Lustre: Skipped 2 previous similar messages [11037.857419] Lustre: server umount lustre-OST0000 complete [11041.775789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11041.778768] LustreError: 373506:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000da6612f9 x1876372830131712/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11041.790449] Lustre: Skipped 1 previous similar message [11056.096268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11056.385675] Lustre: server umount lustre-MDT0000 complete [11057.135641] LustreError: 379711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11057.178745] LustreError: 379711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11060.886987] LustreError: 379696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448893 with bad export cookie 6447104103721560431 [11060.888273] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11060.909820] LustreError: 379696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11067.319424] Lustre: server umount lustre-MDT0001 complete [11080.326506] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [11083.471526] Key type lgssc unregistered [11083.848647] LNet: 384544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11083.855873] LNetError: 384544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11083.880120] LNet: Removed LNI 192.168.206.115@tcp [11084.730875] Key type .llcrypt unregistered [11084.735495] Key type ._llcrypt unregistered [11139.363542] Key type ._llcrypt registered [11139.366840] Key type .llcrypt registered [11139.501836] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [11140.862362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11140.896969] alg: No test for adler32 (adler32-zlib) [11142.068660] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [11142.415803] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [11144.128325] Key type lgssc registered [11145.407917] Lustre: Echo OBD driver; http://www.lustre.org/ [11154.495683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11154.524859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11155.863979] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11155.892900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11156.059142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11161.291790] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11170.481382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11170.560822] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11170.598575] Lustre: 387339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11170.848212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11176.345525] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11184.787456] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11190.928020] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11196.387735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11196.397123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11196.416751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11196.897293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11199.522603] Lustre: server umount lustre-MDT0000 complete [11203.281798] LustreError: 387332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449036 with bad export cookie 13393800620102628757 [11203.293486] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11203.295676] LustreError: 387332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11203.753267] Lustre: server umount lustre-MDT0001 complete [11215.124709] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:10:46 (1789449046) [11222.111549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11222.767225] LustreError: 389851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11222.786119] LustreError: 389851:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11222.842747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11227.709753] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11228.130486] LustreError: 389852:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11233.256361] LustreError: 389851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11235.857132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11236.193061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11240.622933] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11247.989435] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11254.457534] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11261.460380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11261.649443] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11261.862231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11267.846969] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11275.159096] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11282.763316] Lustre: Permanently deactivating lustre-OST0000 [11282.801591] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11291.202147] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11292.826419] LustreError: 392838:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11292.843038] LustreError: 392838:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11292.852367] LustreError: 392838:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11296.000823] LustreError: 392888:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11296.006637] LustreError: 392888:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11304.180829] Lustre: server umount lustre-OST0000 complete [11306.711725] LustreError: 393137:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11306.725932] LustreError: 393137:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11311.073892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11311.077388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11311.081254] Lustre: Skipped 1 previous similar message [11311.098745] Lustre: Skipped 2 previous similar messages [11316.193352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11316.200912] Lustre: Skipped 1 previous similar message [11318.351578] Lustre: server umount lustre-MDT0000 complete [11321.314237] LustreError: 390863:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.335579] LustreError: 390863:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11322.434189] LustreError: 389832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449155 with bad export cookie 13393800620102629338 [11322.440177] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11322.447826] LustreError: 389832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11333.311917] Lustre: server umount lustre-MDT0001 complete [11342.493100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11348.203001] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11351.290503] LustreError: 394298:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11358.059265] Lustre: 394638:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.115@tcp [11360.470731] Lustre: server umount MGS complete [11372.150448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11372.654838] LustreError: 395363:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11372.674932] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11372.684891] Lustre: Skipped 1 previous similar message [11372.734252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11377.210259] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11383.267022] LustreError: 395364:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11383.292990] LustreError: 395364:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11386.694123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11387.052642] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11391.598786] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11399.392594] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11404.787411] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11406.782688] Lustre: Permanently reactivating lustre-OST0000 [11406.793899] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11406.798687] Lustre: Skipped 2 previous similar messages [11432.936317] LustreError: 397098:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11440.588567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11440.833923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11440.838157] Lustre: Skipped 1 previous similar message [11446.797924] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11454.166424] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11463.650503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11463.655099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11463.657898] LustreError: Skipped 1 previous similar message [11463.664602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11463.667413] Lustre: Skipped 1 previous similar message [11463.699126] Lustre: Skipped 1 previous similar message [11468.769441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11468.782079] Lustre: Skipped 1 previous similar message [11474.400485] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11474.653429] Lustre: server umount lustre-OST0000 complete [11479.013533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11479.028746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11479.034682] Lustre: Skipped 3 previous similar messages [11484.941250] Lustre: server umount lustre-MDT0000 complete [11488.778174] LustreError: 395343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449321 with bad export cookie 13393800620102630388 [11488.778843] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11488.810546] LustreError: 395343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11489.256282] Lustre: server umount lustre-MDT0001 complete [11501.313599] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [11504.892965] Key type lgssc unregistered [11505.220835] LNet: 399728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11505.241844] LNetError: 399728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11505.274957] LNet: Removed LNI 192.168.206.115@tcp [11506.061485] Key type .llcrypt unregistered [11506.066341] Key type ._llcrypt unregistered [11524.518231] Key type ._llcrypt registered [11524.522900] Key type .llcrypt registered [11524.643172] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [11538.688734] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [11539.899193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11539.931287] alg: No test for adler32 (adler32-zlib) [11541.029113] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [11541.281768] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [11542.961306] Key type lgssc registered [11544.080703] Lustre: Echo OBD driver; http://www.lustre.org/ [11578.882472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11578.928466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11580.169774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11580.192147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11580.282957] Lustre: lustre-MDT0000: new disk, initializing [11580.382070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11580.405866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11585.899211] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11598.537990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11598.672790] Lustre: 402878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11598.742401] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11598.747154] Lustre: Skipped 1 previous similar message [11598.804487] Lustre: lustre-MDT0001: new disk, initializing [11598.897211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11598.925044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11598.931217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11604.620266] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11615.906484] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11621.930599] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11629.023650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11629.304571] Lustre: lustre-OST0000: new disk, initializing [11629.312655] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11629.321130] Lustre: 404848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11629.418591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11636.594655] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11637.343554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11637.364962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11637.487395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11645.721660] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11650.858713] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11651.111584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11655.090299] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11655.375993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11658.209586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11658.217632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11658.243313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11658.252703] Lustre: Skipped 1 previous similar message [11663.151451] Lustre: server umount lustre-OST0000 complete [11668.453201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11668.455809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11668.465058] Lustre: Skipped 2 previous similar messages [11668.475965] Lustre: Skipped 1 previous similar message [11673.017293] Lustre: server umount lustre-MDT0000 complete [11676.471030] LustreError: 402870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449509 with bad export cookie 15269504611371258718 [11676.480911] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11676.484310] LustreError: 402870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11676.878983] Lustre: server umount lustre-MDT0001 complete [11685.595514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11687.634193] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:18:39 (1789449519) [11695.791947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11696.275513] LustreError: 407308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11696.305648] LustreError: 407308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11696.394525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11700.749033] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11701.730104] LustreError: 407309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11706.882384] LustreError: 407308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11709.094573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11709.494662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11713.701749] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11721.318343] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11727.220638] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11733.948528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11734.300678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11740.352723] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11748.701347] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11751.002259] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11753.790278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11761.125035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11761.139438] LustreError: Skipped 1 previous similar message [11761.148095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11761.163179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11764.242226] Lustre: server umount lustre-OST0000 complete [11770.853107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11770.860967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11770.876086] Lustre: Skipped 1 previous similar message [11770.891469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11770.898043] Lustre: Skipped 2 previous similar messages [11773.846507] Lustre: server umount lustre-MDT0000 complete [11777.505238] LustreError: 407308:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.528311] LustreError: 407308:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11777.606755] LustreError: 407291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449610 with bad export cookie 15269504611371259600 [11777.618890] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11777.623587] LustreError: 407291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11778.177376] Lustre: server umount lustre-MDT0001 complete [11791.037641] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [11793.725102] Key type lgssc unregistered [11794.117939] LNet: 411033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11794.129711] LNetError: 411033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11794.164803] LNet: Removed LNI 192.168.206.115@tcp [11795.085816] Key type .llcrypt unregistered [11795.089213] Key type ._llcrypt unregistered [11818.626101] Key type ._llcrypt registered [11818.627577] Key type .llcrypt registered [11818.734350] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [11818.971370] Key type .llcrypt unregistered [11818.974738] Key type ._llcrypt unregistered [11837.940235] Key type ._llcrypt registered [11837.946635] Key type .llcrypt registered [11838.102458] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [11839.703053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11839.758165] alg: No test for adler32 (adler32-zlib) [11840.733796] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [11841.050225] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [11842.804173] Key type lgssc registered [11844.525442] Lustre: Echo OBD driver; http://www.lustre.org/ [11852.509991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11852.526283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11853.850033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11858.907417] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11864.039237] LustreError: 413240:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.071116] LustreError: 413240:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11867.436903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11867.699341] LustreError: 413239:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11867.715948] LustreError: 413239:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11867.739953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11872.748310] LustreError: 413240:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11872.777720] LustreError: 413240:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11874.990651] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11877.865668] LustreError: 413234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.880743] LustreError: 413234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11878.889098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11882.700647] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11884.005217] LustreError: 413235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11884.017344] LustreError: 413235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11888.514316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11888.854124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11892.605021] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [11902.048442] Lustre: server umount lustre-OST0000 complete [11908.068710] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11908.073123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11908.086618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11909.601766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11909.603357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11914.723690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11914.725855] Lustre: Skipped 1 previous similar message [11915.920885] Lustre: server umount lustre-MDT0000 complete [11919.842832] LustreError: 413235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11919.864565] LustreError: 413235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11920.014076] LustreError: 413221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449752 with bad export cookie 13920903683203079874 [11920.026164] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11920.031649] LustreError: 413221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11929.726242] Lustre: server umount lustre-MDT0001 complete [11940.277927] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [11942.952172] Key type lgssc unregistered [11943.204772] LNet: 416386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11943.218688] LNetError: 416386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11943.243826] LNet: Removed LNI 192.168.206.115@tcp [11944.021341] Key type .llcrypt unregistered [11944.023189] Key type ._llcrypt unregistered [11969.412444] Key type ._llcrypt registered [11969.415576] Key type .llcrypt registered [11969.525683] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [11970.281595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11970.295121] alg: No test for adler32 (adler32-zlib) [11971.394567] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [11971.600585] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [11973.256210] Key type lgssc registered [11974.172399] Lustre: Echo OBD driver; http://www.lustre.org/ [11981.021245] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11981.032562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11996.640657] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12001.760454] LustreError: 417773:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.115@tcp: failed processing log, type 4: rc = -110 [12028.384554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12034.366133] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12041.720915] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12047.966847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12048.475499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12048.534732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12052.318587] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12053.475713] LustreError: 417798:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12053.487611] LustreError: 417798:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12058.596198] LustreError: 417799:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.927962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12060.311576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12063.977841] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12069.847619] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12074.879639] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12084.201854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12084.207954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12084.215835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12087.069847] Lustre: server umount lustre-OST0000 complete [12090.853614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12090.859890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12090.872183] Lustre: Skipped 1 previous similar message [12090.881506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12090.889036] Lustre: Skipped 1 previous similar message [12094.437930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12094.439743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12096.367725] Lustre: server umount lustre-MDT0000 complete [12099.052176] LustreError: 417779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449931 with bad export cookie 2390105793351794438 [12099.058682] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12099.061899] LustreError: 417779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12099.287341] Lustre: server umount lustre-MDT0001 complete [12108.717454] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12110.917799] Key type lgssc unregistered [12111.175917] LNet: 421338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12111.185039] LNetError: 421338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12112.233841] LNet: Removed LNI 192.168.206.115@tcp [12112.898610] Key type .llcrypt unregistered [12112.901871] Key type ._llcrypt unregistered [12133.904835] Key type ._llcrypt registered [12133.909415] Key type .llcrypt registered [12134.006410] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [12134.830959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12134.919464] alg: No test for adler32 (adler32-zlib) [12135.946209] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [12136.213384] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [12137.984187] Key type lgssc registered [12139.076159] Lustre: Echo OBD driver; http://www.lustre.org/ [12146.108710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12146.131638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.508094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12151.238217] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12157.923074] LustreError: 422732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12157.943092] LustreError: 422732:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12158.613154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12158.899184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12162.536851] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12169.413553] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12175.197440] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12181.857422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12182.190489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12188.752353] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12192.436252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12196.278991] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12199.911752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12199.918641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12199.928531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12202.926028] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.15@tcp (stopping) [12202.989588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12203.003845] Lustre: Skipped 1 previous similar message [12206.129781] Lustre: server umount lustre-MDT0000 complete [12208.098297] LustreError: 422732:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12208.114254] LustreError: 422732:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12208.223715] LustreError: 422713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.15@tcp arrived at 1789450041 with bad export cookie 14433355523666791940 [12208.241259] LustreError: 422713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12209.898929] LustreError: 422711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450042 with bad export cookie 14433355523666791044 [12209.919239] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12209.928239] LustreError: 422711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12213.217846] LustreError: 422731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12213.219615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12213.233884] LustreError: 422731:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12213.252445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12213.255420] Lustre: Skipped 3 previous similar messages [12216.140846] Lustre: server umount lustre-OST0000 complete [12219.364381] LustreError: 422731:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12219.382813] LustreError: 422731:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12222.362272] Lustre: server umount lustre-MDT0001 complete [12232.714256] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12235.198481] Key type lgssc unregistered [12235.483412] LNet: 426415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12235.489806] LNetError: 426415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12235.513026] LNet: Removed LNI 192.168.206.115@tcp [12236.365986] Key type .llcrypt unregistered [12236.367784] Key type ._llcrypt unregistered [12258.209756] Key type ._llcrypt registered [12258.211825] Key type .llcrypt registered [12258.298859] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [12259.118539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12259.165621] alg: No test for adler32 (adler32-zlib) [12260.212358] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [12260.464111] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [12262.152205] Key type lgssc registered [12263.514682] Lustre: Echo OBD driver; http://www.lustre.org/ [12271.636960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12271.654505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12273.144310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12277.839247] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12282.341226] LustreError: 427828:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12282.351064] LustreError: 427828:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12286.541629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12286.947495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12290.791175] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12297.235216] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12302.344422] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12308.055343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12308.336155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12314.074094] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12317.549371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12320.350213] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12327.908726] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12327.923197] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12327.927996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12327.947879] Lustre: Skipped 1 previous similar message [12327.959783] Lustre: Skipped 1 previous similar message [12331.338786] Lustre: server umount lustre-MDT0001 complete [12333.042916] LustreError: 427823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12333.071028] LustreError: 427823:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12338.154515] LustreError: 429099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.175410] LustreError: 429099:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12343.267832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12343.268384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12343.269122] LustreError: 428885:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12343.283951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12343.283964] Lustre: Skipped 1 previous similar message [12344.266646] Lustre: server umount lustre-OST0000 complete [12348.087382] Lustre: server umount lustre-MDT0000 complete [12358.966843] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12361.163151] Key type lgssc unregistered [12361.437425] LNet: 431510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12361.442788] LNetError: 431510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12362.470695] LNet: Removed LNI 192.168.206.115@tcp [12363.271643] Key type .llcrypt unregistered [12363.274443] Key type ._llcrypt unregistered [12385.291472] Key type ._llcrypt registered [12385.293375] Key type .llcrypt registered [12385.370474] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [12397.971345] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [12398.899640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12398.945690] alg: No test for adler32 (adler32-zlib) [12399.975380] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [12400.157395] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [12401.824342] Key type lgssc registered [12402.897214] Lustre: Echo OBD driver; http://www.lustre.org/ [12436.852805] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [12450.749921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12450.788169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12451.983706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12452.002849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12452.063490] Lustre: lustre-MDT0000: new disk, initializing [12452.114851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12452.128592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12455.715124] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12468.685754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12468.869973] Lustre: lustre-OST0000: new disk, initializing [12468.873880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12468.878506] Lustre: Skipped 1 previous similar message [12468.886242] Lustre: 436519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12468.956309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12472.875262] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12474.422423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12474.432995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12474.456532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12479.952117] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12492.493676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.596599] Lustre: 435501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12492.658757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12492.755344] Lustre: lustre-MDT0001: new disk, initializing [12492.870959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12492.896137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12492.904736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12496.499108] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12503.758941] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12503.976808] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12507.513415] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12507.743557] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12513.258551] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12513.258869] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12513.272416] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12513.285931] Lustre: Skipped 1 previous similar message [12513.293884] Lustre: Skipped 1 previous similar message [12518.364374] Lustre: server umount lustre-MDT0001 complete [12523.494440] LustreError: 436972:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12523.511510] LustreError: 436972:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12528.617219] LustreError: 435506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12528.635704] LustreError: 435506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12533.730087] LustreError: 438482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12533.749684] LustreError: 438482:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12538.852312] LustreError: 435511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12538.869397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12538.896813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12538.899855] Lustre: Skipped 1 previous similar message [12542.581436] Lustre: server umount lustre-OST0000 complete [12543.971958] LustreError: 438482:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12552.691810] Lustre: server umount lustre-MDT0000 complete [12565.885334] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12568.011502] Key type lgssc unregistered [12568.267488] LNet: 440264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12568.280298] LNetError: 440264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12568.302941] LNet: Removed LNI 192.168.206.115@tcp [12568.952103] Key type .llcrypt unregistered [12568.954503] Key type ._llcrypt unregistered [12605.923512] Key type ._llcrypt registered [12605.926714] Key type .llcrypt registered [12606.048163] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [12606.313278] Key type .llcrypt unregistered [12606.317517] Key type ._llcrypt unregistered [12624.503934] Key type ._llcrypt registered [12624.507875] Key type .llcrypt registered [12624.633272] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [12625.538763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12625.663012] alg: No test for adler32 (adler32-zlib) [12626.703951] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [12627.031298] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [12628.784186] Key type lgssc registered [12630.135360] Lustre: Echo OBD driver; http://www.lustre.org/ [12656.965135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12657.006037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12658.188831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12658.219593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12658.267891] Lustre: lustre-MDT0000: new disk, initializing [12658.334567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12658.347248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12661.695187] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12672.720831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12672.827853] Lustre: 443632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12672.897413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12672.905565] Lustre: Skipped 1 previous similar message [12672.975597] Lustre: lustre-MDT0001: new disk, initializing [12673.109560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12673.152627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12673.162317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12677.529787] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12686.915593] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12692.383457] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12698.994184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12699.143293] Lustre: lustre-OST0000: new disk, initializing [12699.146287] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12699.150905] Lustre: 445600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12699.219248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12704.264431] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12705.349730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12705.358725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12705.469509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12713.300255] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12724.195398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12724.199490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12724.202698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12724.237267] Lustre: Skipped 1 previous similar message [12727.253693] Lustre: server umount lustre-MDT0000 complete [12730.361447] LustreError: 443624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450563 with bad export cookie 5280168772944792166 [12730.368800] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12730.370272] LustreError: 443624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12730.699839] Lustre: server umount lustre-MDT0001 complete [12743.499241] Lustre: server umount lustre-OST0000 complete [12753.052447] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:36:25 (1789450585) [12760.045534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12760.436456] LustreError: 447627:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12760.456458] LustreError: 447627:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12760.510365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12764.394655] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12765.670995] LustreError: 447628:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12770.791738] LustreError: 447627:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12771.808916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12772.127989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12776.526713] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12784.109694] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12789.629949] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12808.552022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12808.769632] Lustre: Found index 0 for lustre-OST0000, updating log [12809.005658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12814.105622] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [12820.716501] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12832.562155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12841.956889] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12841.972302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12841.993734] Lustre: Skipped 1 previous similar message [12842.000252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12842.005992] Lustre: Skipped 2 previous similar messages [12843.049565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12843.059844] Lustre: Skipped 1 previous similar message [12847.142301] Lustre: server umount lustre-OST0000 complete [12862.605291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12862.626037] Lustre: Skipped 1 previous similar message [12862.637697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12862.642885] Lustre: Skipped 2 previous similar messages [12868.065989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12868.074912] Lustre: Skipped 1 previous similar message [12869.126532] LustreError: 448355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12869.144467] LustreError: 448355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12869.255639] Lustre: server umount lustre-MDT0000 complete [12874.962409] LustreError: 447607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450707 with bad export cookie 5280168772944793559 [12874.966994] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12874.974319] LustreError: 447607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12875.349255] Lustre: server umount lustre-MDT0001 complete [12878.948067] LNet: 1 local NIs in recovery (showing 1): 192.168.206.115@tcp [12888.079483] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:38:39 (1789450719) [13504.311821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13505.170355] LustreError: 454061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13505.206051] LustreError: 454061:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13505.298201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13509.694663] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [13510.627529] LustreError: 454062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13515.755375] LustreError: 454061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13517.300687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13517.603775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13522.071664] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [13528.939625] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13534.005803] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13553.459068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13553.640778] LustreError: 455891:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13553.642933] Lustre: Found index 0 for lustre-OST0000, updating log [13553.658950] LustreError: 455891:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13553.973207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13555.872120] LNet: 1 local NIs in recovery (showing 1): 192.168.206.115@tcp [13560.441425] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [13568.134946] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13570.541508] Lustre: server umount lustre-OST0000 complete [13574.113131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13574.125257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13574.136994] Lustre: Skipped 1 previous similar message [13574.144160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13575.201241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13575.206248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13577.320563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13577.323962] Lustre: Skipped 3 previous similar messages [13580.207903] Lustre: server umount lustre-MDT0000 complete [13580.395148] LustreError: 454061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13580.427183] LustreError: 454061:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13583.728954] LustreError: 454042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451416 with bad export cookie 5280168772944794637 [13583.743367] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13583.752780] LustreError: 454042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13584.156254] Lustre: server umount lustre-MDT0001 complete [13613.918991] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [13615.264253] LNet: 1 local NIs in recovery (showing 1): 192.168.206.115@tcp [13623.657318] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [13654.417846] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:51:27 (1789451487) [13673.005498] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [13675.561507] Key type lgssc unregistered [13675.868495] LNet: 460671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13675.880860] LNetError: 460671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13768.046577] LNet: Removed LNI 192.168.206.115@tcp [13768.856741] Key type .llcrypt unregistered [13768.859809] Key type ._llcrypt unregistered [13784.717298] Key type ._llcrypt registered [13784.719902] Key type .llcrypt registered [13784.819377] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [13785.640703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13785.712546] alg: No test for adler32 (adler32-zlib) [13786.791964] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [13787.016663] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [13788.661220] Key type lgssc registered [13789.813296] Lustre: Echo OBD driver; http://www.lustre.org/ [13794.343689] LNet: Added LNI 192.168.206.115@tcp100 [8/256/0/180] [13795.230416] LNet: Added LNI 192.168.206.115@tcp101 [8/256/0/180] [13796.086736] LNet: Added LNI 192.168.206.115@tcp102 [8/256/0/180] [13796.967211] LNet: Added LNI 192.168.206.115@tcp103 [8/256/0/180] [13797.882109] LNet: Added LNI 192.168.206.115@tcp104 [8/256/0/180] [13798.889778] LNet: Added LNI 192.168.206.115@tcp105 [8/256/0/180] [13799.881667] LNet: Added LNI 192.168.206.115@tcp106 [8/256/0/180] [13800.925557] LNet: Added LNI 192.168.206.115@tcp107 [8/256/0/180] [13801.745914] LNet: Added LNI 192.168.206.115@tcp108 [8/256/0/180] [13802.573484] LNet: Added LNI 192.168.206.115@tcp109 [8/256/0/180] [13803.510839] LNet: Added LNI 192.168.206.115@tcp110 [8/256/0/180] [13804.357875] LNet: Added LNI 192.168.206.115@tcp111 [8/256/0/180] [13805.274854] LNet: Added LNI 192.168.206.115@tcp112 [8/256/0/180] [13806.187489] LNet: Added LNI 192.168.206.115@tcp113 [8/256/0/180] [13806.953565] LNet: Added LNI 192.168.206.115@tcp114 [8/256/0/180] [13807.925435] LNet: Added LNI 192.168.206.115@tcp115 [8/256/0/180] [13808.829791] LNet: Added LNI 192.168.206.115@tcp116 [8/256/0/180] [13809.684528] LNet: Added LNI 192.168.206.115@tcp117 [8/256/0/180] [13810.501986] LNet: Added LNI 192.168.206.115@tcp118 [8/256/0/180] [13811.370099] LNet: Added LNI 192.168.206.115@tcp119 [8/256/0/180] [13812.118300] LNet: Added LNI 192.168.206.115@tcp120 [8/256/0/180] [13812.902441] LNet: Added LNI 192.168.206.115@tcp121 [8/256/0/180] [13813.624388] LNet: Added LNI 192.168.206.115@tcp122 [8/256/0/180] [13814.430758] LNet: Added LNI 192.168.206.115@tcp123 [8/256/0/180] [13815.174122] LNet: Added LNI 192.168.206.115@tcp124 [8/256/0/180] [13815.887695] LNet: Added LNI 192.168.206.115@tcp125 [8/256/0/180] [13816.630668] LNet: Added LNI 192.168.206.115@tcp126 [8/256/0/180] [13817.302918] LNet: Added LNI 192.168.206.115@tcp127 [8/256/0/180] [13818.162659] LNet: Added LNI 192.168.206.115@tcp128 [8/256/0/180] [13818.991510] LNet: Added LNI 192.168.206.115@tcp129 [8/256/0/180] [13819.739578] LNet: Added LNI 192.168.206.115@tcp130 [8/256/0/180] [13820.510396] LNet: Added LNI 192.168.206.115@tcp131 [8/256/0/180] [13821.577659] LNet: Added LNI 192.168.206.115@tcp132 [8/256/0/180] [13822.489589] LNet: Added LNI 192.168.206.115@tcp133 [8/256/0/180] [13823.480513] LNet: Added LNI 192.168.206.115@tcp134 [8/256/0/180] [13824.411902] LNet: Added LNI 192.168.206.115@tcp135 [8/256/0/180] [13831.497825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13831.520572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13832.642021] Lustre: 463696:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [13832.790744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13832.821695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13832.911164] Lustre: lustre-MDT0000: new disk, initializing [13833.003616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13833.016568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13837.689484] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [13850.008278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13850.227135] Lustre: 463713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13850.307398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13850.313340] Lustre: Skipped 1 previous similar message [13850.473331] Lustre: lustre-MDT0001: new disk, initializing [13850.541331] Lustre: 464514:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [13850.556397] Lustre: 464514:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13850.659434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13850.715716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13850.730635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13855.481868] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [13858.862215] Lustre: 464898:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [13858.883442] Lustre: 464898:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13865.102283] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13870.661571] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13878.479269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13878.953875] Lustre: lustre-OST0000: new disk, initializing [13878.959255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13878.967909] Lustre: 465687:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13879.116385] Lustre: 465692:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [13879.123869] Lustre: 465692:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13879.187163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13885.882547] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [13888.053925] Lustre: 466128:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.115@tcp: can't add NID 192.168.206.115@tcp130: rc = -75 [13888.059992] Lustre: 466128:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13888.126815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13888.139849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13888.342193] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13896.866259] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13908.963632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13908.966046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13908.990575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13909.001575] Lustre: Skipped 1 previous similar message [13909.028076] Lustre: Skipped 1 previous similar message [13910.355803] Lustre: server umount lustre-OST0000 complete [13917.155304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13917.159767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13917.185948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13919.205674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13919.206404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13920.925480] Lustre: server umount lustre-MDT0000 complete [13925.231806] LustreError: 464513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451758 with bad export cookie 9713985027675591727 [13925.244085] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13925.264734] LustreError: 464513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13925.725646] Lustre: server umount lustre-MDT0001 complete [13938.465346] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [13941.682396] Key type lgssc unregistered [13942.138438] LNet: 467599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13942.146551] LNetError: 467599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13943.200660] LNet: Removed LNI 192.168.206.115@tcp [13943.202701] LNetError: 467599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13943.222191] LNet: Removed LNI 192.168.206.115@tcp100 [13944.288502] LNet: Removed LNI 192.168.206.115@tcp101 [13944.294066] LNetError: 467599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13944.319635] LNetError: 467599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13944.346898] LNet: Removed LNI 192.168.206.115@tcp102 [13945.440138] LNet: Removed LNI 192.168.206.115@tcp103 [13946.464148] LNet: Removed LNI 192.168.206.115@tcp104 [13946.468385] LNetError: 467599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13946.491554] LNetError: 467599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13946.506105] LNet: Removed LNI 192.168.206.115@tcp105 [13947.552138] LNet: Removed LNI 192.168.206.115@tcp106 [13948.576243] LNet: Removed LNI 192.168.206.115@tcp107 [13949.600184] LNet: Removed LNI 192.168.206.115@tcp108 [13950.624386] LNet: Removed LNI 192.168.206.115@tcp109 [13950.636393] LNetError: 467599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13950.653054] LNetError: 467599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13950.667262] LNet: Removed LNI 192.168.206.115@tcp110 [13951.712551] LNet: Removed LNI 192.168.206.115@tcp111 [13952.736206] LNet: Removed LNI 192.168.206.115@tcp112 [13952.738200] LNet: Removed LNI 192.168.206.115@tcp113 [13952.750346] LNet: Removed LNI 192.168.206.115@tcp114 [13952.768138] LNet: Removed LNI 192.168.206.115@tcp115 [13952.777523] LNet: Removed LNI 192.168.206.115@tcp116 [13952.783685] LNet: Removed LNI 192.168.206.115@tcp117 [13952.789526] LNet: Removed LNI 192.168.206.115@tcp118 [13952.791363] LNet: Removed LNI 192.168.206.115@tcp119 [13952.793345] LNet: Removed LNI 192.168.206.115@tcp120 [13952.795219] LNet: Removed LNI 192.168.206.115@tcp121 [13952.797202] LNet: Removed LNI 192.168.206.115@tcp122 [13952.799123] LNet: Removed LNI 192.168.206.115@tcp123 [13952.801512] LNet: Removed LNI 192.168.206.115@tcp124 [13952.804557] LNet: Removed LNI 192.168.206.115@tcp125 [13952.808239] LNet: Removed LNI 192.168.206.115@tcp126 [13952.811945] LNet: Removed LNI 192.168.206.115@tcp127 [13952.814924] LNet: Removed LNI 192.168.206.115@tcp128 [13952.817379] LNet: Removed LNI 192.168.206.115@tcp129 [13952.820039] LNet: Removed LNI 192.168.206.115@tcp130 [13952.822160] LNet: Removed LNI 192.168.206.115@tcp131 [13952.825454] LNet: Removed LNI 192.168.206.115@tcp132 [13952.827875] LNet: Removed LNI 192.168.206.115@tcp133 [13952.829955] LNet: Removed LNI 192.168.206.115@tcp134 [13952.841055] LNet: Removed LNI 192.168.206.115@tcp135 [13954.112150] Key type .llcrypt unregistered [13954.117475] Key type ._llcrypt unregistered [13981.992176] Key type ._llcrypt registered [13981.994641] Key type .llcrypt registered [13982.130417] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [13997.203832] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [13998.707107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13998.796134] alg: No test for adler32 (adler32-zlib) [13999.937931] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [14000.215939] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [14001.952440] Key type lgssc registered [14003.350723] Lustre: Echo OBD driver; http://www.lustre.org/ [14037.730183] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:57:50 (1789451870) [14057.973922] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [14060.953980] Key type lgssc unregistered [14061.322337] LNet: 471690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14061.328762] LNetError: 471690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14061.343886] LNet: Removed LNI 192.168.206.115@tcp [14062.584359] Key type .llcrypt unregistered [14062.587421] Key type ._llcrypt unregistered [14079.105830] Key type ._llcrypt registered [14079.108875] Key type .llcrypt registered [14079.218909] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [14080.611786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14080.691592] alg: No test for adler32 (adler32-zlib) [14081.739734] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [14082.097620] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [14083.832321] Key type lgssc registered [14085.063167] Lustre: Echo OBD driver; http://www.lustre.org/ [14094.444881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14094.474177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14095.842920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14095.884853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14095.986388] Lustre: lustre-MDT0000: new disk, initializing [14096.035257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14096.061923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14101.146717] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14110.576215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14110.720579] Lustre: 472899:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14110.805543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14110.810911] Lustre: Skipped 1 previous similar message [14110.940193] Lustre: lustre-MDT0001: new disk, initializing [14111.025391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14111.052964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14111.057778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14114.729992] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14124.777920] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14131.190987] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14138.589469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14138.831758] Lustre: lustre-OST0000: new disk, initializing [14138.836877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14138.849865] Lustre: 474869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14138.973140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14145.664763] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14147.252185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14147.265413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14147.389757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14155.510313] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14160.300609] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing lnet_if_list [14166.587271] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14166.662678] LNet: Added LNI 192.168.206.115@tcp10 [8/256/0/180] [14173.590837] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14173.649230] LNet: Added LNI 192.168.206.115@tcp11 [8/256/0/180] [14178.109507] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14178.137148] LNet: Added LNI 192.168.206.115@tcp12 [8/256/0/180] [14182.807131] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14182.839540] LNet: Added LNI 192.168.206.115@tcp13 [8/256/0/180] [14187.467574] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14187.511226] LNet: Added LNI 192.168.206.115@tcp14 [8/256/0/180] [14192.630051] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14192.657549] LNet: Added LNI 192.168.206.115@tcp15 [8/256/0/180] [14197.537125] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14197.567464] LNet: Added LNI 192.168.206.115@tcp16 [8/256/0/180] [14202.236218] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14202.260081] LNet: Added LNI 192.168.206.115@tcp17 [8/256/0/180] [14207.061196] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14207.094914] LNet: Added LNI 192.168.206.115@tcp18 [8/256/0/180] [14212.359903] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14212.384736] LNet: Added LNI 192.168.206.115@tcp19 [8/256/0/180] [14217.579105] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14217.606747] LNet: Added LNI 192.168.206.115@tcp20 [8/256/0/180] [14222.730675] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14222.771921] LNet: Added LNI 192.168.206.115@tcp21 [8/256/0/180] [14228.845243] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14228.919752] LNet: Added LNI 192.168.206.115@tcp22 [8/256/0/180] [14234.639894] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14234.663653] LNet: Added LNI 192.168.206.115@tcp23 [8/256/0/180] [14239.732771] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14239.758305] LNet: Added LNI 192.168.206.115@tcp24 [8/256/0/180] [14245.974368] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14246.019571] LNet: Added LNI 192.168.206.115@tcp25 [8/256/0/180] [14251.270210] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14251.296543] LNet: Added LNI 192.168.206.115@tcp26 [8/256/0/180] [14257.508264] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14257.548542] LNet: Added LNI 192.168.206.115@tcp27 [8/256/0/180] [14262.956896] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14262.995631] LNet: Added LNI 192.168.206.115@tcp28 [8/256/0/180] [14268.454795] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14268.477214] LNet: Added LNI 192.168.206.115@tcp29 [8/256/0/180] [14274.606216] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14274.633679] LNet: Added LNI 192.168.206.115@tcp30 [8/256/0/180] [14281.002554] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14281.035946] LNet: Added LNI 192.168.206.115@tcp31 [8/256/0/180] [14286.786225] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14286.805486] LNet: Added LNI 192.168.206.115@tcp32 [8/256/0/180] [14291.206390] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14291.243514] LNet: Added LNI 192.168.206.115@tcp33 [8/256/0/180] [14296.368864] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14296.392939] LNet: Added LNI 192.168.206.115@tcp34 [8/256/0/180] [14303.549222] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14303.593697] LNet: Added LNI 192.168.206.115@tcp35 [8/256/0/180] [14312.359608] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14312.404717] LNet: Added LNI 192.168.206.115@tcp36 [8/256/0/180] [14319.568987] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14319.601090] LNet: Added LNI 192.168.206.115@tcp37 [8/256/0/180] [14326.415346] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14326.446296] LNet: Added LNI 192.168.206.115@tcp38 [8/256/0/180] [14333.316170] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14333.341511] LNet: Added LNI 192.168.206.115@tcp39 [8/256/0/180] [14340.267661] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14340.295037] LNet: Added LNI 192.168.206.115@tcp40 [8/256/0/180] [14346.968584] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14346.998550] LNet: Added LNI 192.168.206.115@tcp41 [8/256/0/180] [14352.793342] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14352.830087] LNet: Added LNI 192.168.206.115@tcp42 [8/256/0/180] [14359.735847] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14359.771333] LNet: Added LNI 192.168.206.115@tcp43 [8/256/0/180] [14367.120556] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14367.153241] LNet: Added LNI 192.168.206.115@tcp44 [8/256/0/180] [14374.623695] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14374.645536] LNet: Added LNI 192.168.206.115@tcp45 [8/256/0/180] [14381.356139] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14381.390341] LNet: Added LNI 192.168.206.115@tcp46 [8/256/0/180] [14387.905107] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14387.932966] LNet: Added LNI 192.168.206.115@tcp47 [8/256/0/180] [14394.849515] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14394.880844] LNet: Added LNI 192.168.206.115@tcp48 [8/256/0/180] [14401.933314] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14401.954693] LNet: Added LNI 192.168.206.115@tcp49 [8/256/0/180] [14409.865049] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14409.916697] LNet: Added LNI 192.168.206.115@tcp50 [8/256/0/180] [14425.010495] Lustre: server umount lustre-MDT0000 complete [14427.997191] LustreError: 472892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452260 with bad export cookie 2327764035740326874 [14428.005857] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14428.451607] Lustre: server umount lustre-MDT0001 complete [14444.512207] Lustre: 472171:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789452261/real 1789452262] req@0000000084e3c587 x1876376330331392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789452277 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14444.538593] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14445.150283] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [14448.608214] Lustre: 472173:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789452265/real 1789452265] req@000000000be9ec85 x1876376330331520/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789452281 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14448.641015] Lustre: 472173:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [14453.411302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14453.729501] Lustre: 472172:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789452270/real 1789452270] req@0000000064adbb61 x1876376330331904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789452286 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14453.765430] Lustre: 472172:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14458.848178] Lustre: 472174:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789452275/real 1789452275] req@00000000952afef0 x1876376330332288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789452291 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14458.880689] Lustre: 472174:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14472.530174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14477.240294] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14480.876599] LustreError: 474864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14480.899267] LustreError: 474864:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [14484.992125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14485.360862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14489.468925] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14490.127358] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14496.946886] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14502.042738] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14523.877457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14523.884670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14523.893729] Lustre: Skipped 1 previous similar message [14523.898681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14525.414606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14525.417644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14528.198476] Lustre: server umount lustre-OST0000 complete [14535.652644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14535.684794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14535.692385] Lustre: Skipped 2 previous similar messages [14538.640134] Lustre: server umount lustre-MDT0000 complete [14540.775540] LustreError: 500706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14540.800607] LustreError: 500706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14542.503129] LustreError: 473695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452375 with bad export cookie 2327764035740329443 [14542.514116] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14542.522244] LustreError: 473695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14542.912461] Lustre: server umount lustre-MDT0001 complete [14553.246850] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [14555.579452] Key type lgssc unregistered [14555.940841] LNet: 501920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14555.949479] LNetError: 501920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14555.954595] LNet: Removed LNI 192.168.206.115@tcp [14555.956837] LNet: Removed LNI 192.168.206.115@tcp10 [14555.958692] LNet: Removed LNI 192.168.206.115@tcp11 [14555.960503] LNet: Removed LNI 192.168.206.115@tcp12 [14555.964175] LNet: Removed LNI 192.168.206.115@tcp13 [14555.966521] LNet: Removed LNI 192.168.206.115@tcp14 [14555.968572] LNet: Removed LNI 192.168.206.115@tcp15 [14555.970732] LNet: Removed LNI 192.168.206.115@tcp16 [14555.973073] LNet: Removed LNI 192.168.206.115@tcp17 [14555.975477] LNet: Removed LNI 192.168.206.115@tcp18 [14555.977443] LNet: Removed LNI 192.168.206.115@tcp19 [14555.981039] LNet: Removed LNI 192.168.206.115@tcp20 [14555.983115] LNet: Removed LNI 192.168.206.115@tcp21 [14555.985385] LNet: Removed LNI 192.168.206.115@tcp22 [14555.987614] LNet: Removed LNI 192.168.206.115@tcp23 [14555.989856] LNet: Removed LNI 192.168.206.115@tcp24 [14555.994173] LNet: Removed LNI 192.168.206.115@tcp25 [14555.996815] LNet: Removed LNI 192.168.206.115@tcp26 [14556.001707] LNet: Removed LNI 192.168.206.115@tcp27 [14556.004903] LNet: Removed LNI 192.168.206.115@tcp28 [14556.008206] LNet: Removed LNI 192.168.206.115@tcp29 [14556.011629] LNet: Removed LNI 192.168.206.115@tcp30 [14556.014582] LNet: Removed LNI 192.168.206.115@tcp31 [14556.020769] LNet: Removed LNI 192.168.206.115@tcp32 [14556.025640] LNet: Removed LNI 192.168.206.115@tcp33 [14556.030746] LNet: Removed LNI 192.168.206.115@tcp34 [14556.033513] LNet: Removed LNI 192.168.206.115@tcp35 [14556.036822] LNet: Removed LNI 192.168.206.115@tcp36 [14556.039645] LNet: Removed LNI 192.168.206.115@tcp37 [14556.042826] LNet: Removed LNI 192.168.206.115@tcp38 [14556.045432] LNet: Removed LNI 192.168.206.115@tcp39 [14556.048456] LNet: Removed LNI 192.168.206.115@tcp40 [14556.051243] LNet: Removed LNI 192.168.206.115@tcp41 [14556.053960] LNet: Removed LNI 192.168.206.115@tcp42 [14556.056932] LNet: Removed LNI 192.168.206.115@tcp43 [14556.060134] LNet: Removed LNI 192.168.206.115@tcp44 [14556.063263] LNet: Removed LNI 192.168.206.115@tcp45 [14556.066255] LNet: Removed LNI 192.168.206.115@tcp46 [14556.070704] LNet: Removed LNI 192.168.206.115@tcp47 [14556.073870] LNet: Removed LNI 192.168.206.115@tcp48 [14556.077264] LNet: Removed LNI 192.168.206.115@tcp49 [14556.085791] LNet: Removed LNI 192.168.206.115@tcp50 [14556.683773] Key type .llcrypt unregistered [14556.685304] Key type ._llcrypt unregistered [14578.875711] Key type ._llcrypt registered [14578.889967] Key type .llcrypt registered [14579.044652] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [14593.633088] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [14594.224109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14594.319830] alg: No test for adler32 (adler32-zlib) [14595.325369] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [14595.570787] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [14597.240217] Key type lgssc registered [14598.330149] Lustre: Echo OBD driver; http://www.lustre.org/ [14632.169040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14632.191773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14633.490197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14633.536393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14633.595391] Lustre: lustre-MDT0000: new disk, initializing [14633.664887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14633.682956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14637.444935] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14647.322889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14647.428858] Lustre: 505111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14647.513909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14647.518655] Lustre: Skipped 1 previous similar message [14647.587083] Lustre: lustre-MDT0001: new disk, initializing [14647.664003] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14647.688610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14647.695708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14651.554976] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14661.817490] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14668.421912] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14676.328423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14676.550429] Lustre: *** cfs_fail_loc=90f, val=0*** [14676.630248] Lustre: lustre-OST0000: new disk, initializing [14676.640257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14676.650602] Lustre: 507128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14676.734733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14682.680472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14682.691646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14682.799532] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14682.807169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14692.182676] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14703.594391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14703.596259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14703.598792] LustreError: Skipped 1 previous similar message [14703.604112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14703.618735] Lustre: Skipped 1 previous similar message [14703.646413] Lustre: Skipped 1 previous similar message [14708.710299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14708.715727] Lustre: Skipped 1 previous similar message [14713.827465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14713.844033] Lustre: Skipped 1 previous similar message [14714.336113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14714.570689] Lustre: server umount lustre-OST0000 complete [14718.945580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14718.955509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14718.959449] Lustre: Skipped 1 previous similar message [14718.980352] Lustre: Skipped 1 previous similar message [14724.065559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14724.082562] Lustre: Skipped 1 previous similar message [14724.521716] Lustre: server umount lustre-MDT0000 complete [14727.948612] LustreError: 507127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452560 with bad export cookie 9094164003626040574 [14727.954508] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14727.957302] LustreError: 507127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14728.358155] Lustre: server umount lustre-MDT0001 complete [14739.156926] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [14742.022847] Key type lgssc unregistered [14742.369711] LNet: 509030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14742.377653] LNetError: 509030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14742.400872] LNet: Removed LNI 192.168.206.115@tcp [14743.177495] Key type .llcrypt unregistered [14743.181134] Key type ._llcrypt unregistered [14761.890445] Key type ._llcrypt registered [14761.894821] Key type .llcrypt registered [14762.018181] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [14763.110675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14763.123463] alg: No test for adler32 (adler32-zlib) [14764.195234] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [14764.380400] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [14766.017750] Key type lgssc registered [14767.269852] Lustre: Echo OBD driver; http://www.lustre.org/ [14774.340520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14774.360409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14775.786437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14780.377378] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14786.024401] LustreError: 510298:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14786.042094] LustreError: 510298:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14789.178175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14789.646589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14794.395824] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14803.293548] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14809.857895] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14816.080570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14816.402691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14822.266215] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14831.023329] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14845.480918] loop0: detected capacity change from 0 to 204800000 [14845.714764] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14845.742257] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14845.751994] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14852.434925] loop0: detected capacity change from 0 to 204800000 [14852.644569] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14852.662928] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14852.689128] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14857.820147] loop0: detected capacity change from 0 to 204800000 [14864.279559] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14864.418124] Lustre: fs15246-OST0000: new disk, initializing [14864.423447] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14864.436551] Lustre: 513892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14864.489796] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14870.729348] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14881.223502] loop1: detected capacity change from 0 to 204800000 [14887.643804] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14887.819578] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14887.906226] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14887.976370] Lustre: fs15246-MDT0000: new disk, initializing [14888.131927] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14888.166513] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14888.323793] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14889.377081] Lustre: 515017:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006fd40a18 x1876377046401408/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789452738 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14892.161114] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [14898.475567] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14898.492772] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14898.516381] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14915.557441] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14915.573045] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14920.216897] Lustre: server umount fs15246-MDT0000 complete [14920.681301] LustreError: 512139:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14925.796777] LustreError: 514695:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14930.775107] Lustre: server umount fs15246-OST0000 complete [14946.278234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14946.288075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14946.300055] Lustre: Skipped 1 previous similar message [14946.325152] Lustre: Skipped 1 previous similar message [14950.345349] Lustre: server umount lustre-OST0000 complete [14956.532807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14956.550408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14956.555812] Lustre: Skipped 1 previous similar message [14956.569303] Lustre: Skipped 1 previous similar message [14961.636536] LustreError: 510295:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14961.658781] LustreError: 510295:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14961.789422] Lustre: server umount lustre-MDT0000 complete [14965.917500] LustreError: 510279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452798 with bad export cookie 7093159992119166721 [14965.918716] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14965.934139] LustreError: 510279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14966.402639] Lustre: server umount lustre-MDT0001 complete [14976.282117] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [14978.700268] Key type lgssc unregistered [14979.011592] LNet: 518111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14979.018910] LNetError: 518111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14979.036173] LNet: Removed LNI 192.168.206.115@tcp [14979.876447] Key type .llcrypt unregistered [14979.883037] Key type ._llcrypt unregistered [14999.494068] Key type ._llcrypt registered [14999.497981] Key type .llcrypt registered [14999.609279] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [15000.471854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15000.558615] alg: No test for adler32 (adler32-zlib) [15001.589803] Lustre: Lustre: Build Version: 2.17.58_39_g975873d [15001.763692] LNet: Added LNI 192.168.206.115@tcp [8/256/0/180] [15003.376185] Key type lgssc registered [15004.501871] Lustre: Echo OBD driver; http://www.lustre.org/ [15045.539503] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [15056.290943] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [15094.865159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15094.893165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15096.113272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15096.134895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15096.238499] Lustre: lustre-MDT0000: new disk, initializing [15096.328199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15096.355400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15100.956155] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [15113.646353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15113.789787] Lustre: 522581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15113.879900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15113.885863] Lustre: Skipped 1 previous similar message [15113.983551] Lustre: lustre-MDT0001: new disk, initializing [15114.193973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15114.253323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15114.279589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15119.118097] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [15129.161350] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15134.358961] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15140.679865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15140.956133] Lustre: lustre-OST0000: new disk, initializing [15140.960359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15140.966749] Lustre: 524548:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15141.041645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15146.892319] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_default_debug -1 all [15148.631603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15148.651460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15148.774066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15157.056956] Lustre: DEBUG MARKER: oleg615-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15162.299953] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15162.586520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15168.138617] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15168.416893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15174.626709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15174.645328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15174.663596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15174.665423] Lustre: Skipped 1 previous similar message [15174.683587] Lustre: Skipped 1 previous similar message [15176.589549] Lustre: server umount lustre-OST0000 complete [15180.772310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15180.782916] LustreError: Skipped 1 previous similar message [15180.787050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15180.801979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15184.866176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15184.867540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15186.891613] Lustre: server umount lustre-MDT0000 complete [15190.716949] LustreError: 522571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453023 with bad export cookie 8206078782213807193 [15190.730626] LustreError: MGC192.168.206.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15190.730801] LustreError: 522571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15191.095508] Lustre: server umount lustre-MDT0001 complete [15217.822874] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing set_hostid [15228.706503] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing load_modules_local [15265.645442] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14971 sec ========= 02:18:17 (1789453097) [15267.794664] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:18:19 (1789453099) === [15271.857338] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:18:24 (1789453104) === [15310.002555] Lustre: DEBUG MARKER: oleg615-server.virtnet: executing unload_modules_local [15312.393381] Key type lgssc unregistered [15312.680415] LNet: 531193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15312.698132] LNetError: 531193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15312.713319] LNet: Removed LNI 192.168.206.115@tcp [15313.504717] Key type .llcrypt unregistered [15313.508382] Key type ._llcrypt unregistered