[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512319120 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003274] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.011048] LSM: Security Framework initializing [ 0.012064] Yama: becoming mindful. [ 0.013063] SELinux: Initializing. [ 0.014096] *** VALIDATE selinux *** [ 0.023867] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028751] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030103] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032135] *** VALIDATE tmpfs *** [ 0.034463] *** VALIDATE proc *** [ 0.035269] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042007] Spectre V2 : User space: Vulnerable [ 0.043011] Speculative Store Bypass: Vulnerable [ 0.046550] debug: unmapping init [mem 0xffffffff9ac59000-0xffffffff9ac60fff] [ 0.047622] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049331] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050025] ... version: 2 [ 0.051015] ... bit width: 48 [ 0.052014] ... generic registers: 4 [ 0.053014] ... value mask: 0000ffffffffffff [ 0.054016] ... max period: 00007fffffffffff [ 0.055019] ... fixed-purpose events: 3 [ 0.056015] ... event mask: 000000070000000f [ 0.057305] rcu: Hierarchical SRCU implementation. [ 0.059521] smp: Bringing up secondary CPUs ... [ 0.060640] x86: Booting SMP configuration: [ 0.061031] .... node #0, CPUs: #1 #2 #3 [ 0.065151] smp: Brought up 1 node, 4 CPUs [ 0.067016] smpboot: Max logical packages: 1 [ 0.068014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.166865] node 0 deferred pages initialised in 95ms [ 0.170136] devtmpfs: initialized [ 0.172288] x86/mm: Memory block size: 128MB [ 0.175930] gcov: version magic: 0x41383552 [ 0.178296] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.182088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.185324] pinctrl core: initialized pinctrl subsystem [ 0.188259] [ 0.188877] ************************************************************* [ 0.191016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194017] ** ** [ 0.196013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.199076] ** ** [ 0.201013] ** This means that this kernel is built to expose internal ** [ 0.204015] ** IOMMU data structures, which may compromise security on ** [ 0.207018] ** your system. ** [ 0.209012] ** ** [ 0.212067] ** If you see this message and you are not debugging the ** [ 0.214016] ** kernel, report this immediately to your vendor! ** [ 0.217016] ** ** [ 0.220013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.222012] ************************************************************* [ 0.225666] NET: Registered protocol family 16 [ 0.227419] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.229049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.232085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.236461] cpuidle: using governor menu [ 0.238262] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.239577] PCI: Using configuration type 1 for base access [ 0.241182] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.250073] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.251000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.254232] cryptd: max_cpu_qlen set to 1000 [ 0.258271] ACPI: Added _OSI(Module Device) [ 0.260013] ACPI: Added _OSI(Processor Device) [ 0.262017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.263017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.268138] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.275637] ACPI: Interpreter enabled [ 0.277065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.279021] ACPI: Using IOAPIC for interrupt routing [ 0.280098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.283414] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.294626] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.297046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.299019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.303092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.309064] acpiphp: Slot [2] registered [ 0.310215] acpiphp: Slot [5] registered [ 0.312167] acpiphp: Slot [6] registered [ 0.314149] acpiphp: Slot [7] registered [ 0.315125] acpiphp: Slot [8] registered [ 0.316136] acpiphp: Slot [9] registered [ 0.318130] acpiphp: Slot [10] registered [ 0.320014] acpiphp: Slot [3] registered [ 0.321098] acpiphp: Slot [4] registered [ 0.323138] acpiphp: Slot [11] registered [ 0.324088] acpiphp: Slot [12] registered [ 0.325093] acpiphp: Slot [13] registered [ 0.326088] acpiphp: Slot [14] registered [ 0.327087] acpiphp: Slot [15] registered [ 0.328080] acpiphp: Slot [16] registered [ 0.329088] acpiphp: Slot [17] registered [ 0.331120] acpiphp: Slot [18] registered [ 0.332211] acpiphp: Slot [19] registered [ 0.334107] acpiphp: Slot [20] registered [ 0.335100] acpiphp: Slot [21] registered [ 0.337120] acpiphp: Slot [22] registered [ 0.338119] acpiphp: Slot [23] registered [ 0.340173] acpiphp: Slot [24] registered [ 0.342155] acpiphp: Slot [25] registered [ 0.343162] acpiphp: Slot [26] registered [ 0.345110] acpiphp: Slot [27] registered [ 0.347129] acpiphp: Slot [28] registered [ 0.348130] acpiphp: Slot [29] registered [ 0.349100] acpiphp: Slot [30] registered [ 0.351136] acpiphp: Slot [31] registered [ 0.353074] PCI host bridge to bus 0000:00 [ 0.354019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.357025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.359024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.362027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.367027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.369196] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.372420] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.375295] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.385016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.391643] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.394025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.397020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.399021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.402554] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.405079] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.407039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.410073] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.416019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.429029] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.435023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.441068] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.449023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.458024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.475024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.486790] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.501062] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.513022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.533022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.546391] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.555023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.563024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.584024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.600995] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.614023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.622025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.644023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.653472] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.659040] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.668030] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.690022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.701000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.708021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.725023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.763025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.784069] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.786409] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.788370] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.791428] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.793272] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.800040] iommu: Default domain type: Passthrough [ 0.802388] SCSI subsystem initialized [ 0.804245] ACPI: bus type USB registered [ 0.805164] usbcore: registered new interface driver usbfs [ 0.807121] usbcore: registered new interface driver hub [ 0.809114] usbcore: registered new device driver usb [ 0.811225] pps_core: LinuxPPS API ver. 1 registered [ 0.812013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.816089] PTP clock support registered [ 0.818092] EDAC MC: Ver: 3.0.0 [ 0.821173] PCI: Using ACPI for IRQ routing [ 0.823030] NetLabel: Initializing [ 0.824013] NetLabel: domain hash size = 128 [ 0.826016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.828116] NetLabel: unlabeled traffic allowed by default [ 0.829145] vgaarb: loaded [ 0.830225] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.832019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.837005] clocksource: Switched to clocksource kvm-clock [ 0.950928] VFS: Disk quotas dquot_6.6.0 [ 0.952696] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.955447] *** VALIDATE ramfs *** [ 0.956771] *** VALIDATE hugetlbfs *** [ 0.958432] pnp: PnP ACPI init [ 0.960947] pnp: PnP ACPI: found 6 devices [ 0.991029] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.994038] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.996233] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.998423] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.000629] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.003115] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.006165] NET: Registered protocol family 2 [ 1.008784] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.013941] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.017652] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.022955] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.026659] TCP: Hash tables configured (established 65536 bind 65536) [ 1.029718] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.032905] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.036072] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.039133] NET: Registered protocol family 1 [ 1.043153] RPC: Registered named UNIX socket transport module. [ 1.045620] RPC: Registered udp transport module. [ 1.047353] RPC: Registered tcp transport module. [ 1.049209] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.052369] NET: Registered protocol family 44 [ 1.054211] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.056286] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.058745] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.061594] PCI: CLS 0 bytes, default 64 [ 1.063677] Unpacking initramfs... [ 2.524428] debug: unmapping init [mem 0xffff9c507cc54000-0xffff9c507ffbffff] [ 2.531068] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.533407] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.537945] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.035874] Initialise system trusted keyrings [ 3.037657] Key type blacklist registered [ 3.041257] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.050575] zbud: loaded [ 3.055042] *** VALIDATE nfs *** [ 3.056473] *** VALIDATE nfs4 *** [ 3.058064] pstore: using deflate compression [ 3.061902] Platform Keyring initialized [ 3.172265] NET: Registered protocol family 38 [ 3.174214] Key type asymmetric registered [ 3.175838] Asymmetric key parser 'x509' registered [ 3.177880] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.180983] io scheduler mq-deadline registered [ 3.182689] io scheduler kyber registered [ 3.183870] io scheduler bfq registered [ 3.185883] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.188715] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.191635] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.194531] ACPI: Power Button [PWRF] [ 3.199821] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.207780] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.227174] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.238391] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.261908] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.292781] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.335545] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.340918] Non-volatile memory driver v1.3 [ 3.342747] Linux agpgart interface v0.103 [ 3.377409] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.380431] vda: detected capacity change from 0 to 74698752 [ 3.401635] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.405034] vdb: detected capacity change from 0 to 1073741824 [ 3.426800] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.430210] vdc: detected capacity change from 0 to 2621440000 [ 3.448320] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.451070] vdd: detected capacity change from 0 to 2621440000 [ 3.471768] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.475035] vde: detected capacity change from 0 to 4294967296 [ 3.501172] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.505644] vdf: detected capacity change from 0 to 4294967296 [ 3.514409] libphy: Fixed MDIO Bus: probed [ 3.526508] usbcore: registered new interface driver usbserial_generic [ 3.529089] usbserial: USB Serial support registered for generic [ 3.531248] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.535553] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.537539] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.539778] mousedev: PS/2 mouse device common for all mice [ 3.542754] rtc_cmos 00:05: RTC can wake from S4 [ 3.545990] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.549366] rtc_cmos 00:05: registered as rtc0 [ 3.551366] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.553398] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.555517] intel_pstate: CPU model not supported [ 3.559096] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.563980] hid: raw HID events driver (C) Jiri Kosina [ 3.566387] usbcore: registered new interface driver usbhid [ 3.568479] usbhid: USB HID core driver [ 3.570089] drop_monitor: Initializing network drop monitor service [ 3.572606] Initializing XFRM netlink socket [ 3.574655] NET: Registered protocol family 10 [ 3.577478] Segment Routing with IPv6 [ 3.578531] NET: Registered protocol family 17 [ 3.580299] mpls_gso: MPLS GSO support [ 3.585291] RAS: Correctable Errors collector initialized. [ 3.587600] AVX version of gcm_enc/dec engaged. [ 3.589307] AES CTR mode by8 optimization enabled [ 3.665991] sched_clock: Marking stable (3665954190, 0)->(4619562001, -953607811) [ 3.669875] registered taskstats version 1 [ 3.672135] Loading compiled-in X.509 certificates [ 3.674420] zswap: loaded using pool lzo/zbud [ 3.699102] Key type big_key registered [ 3.711216] Key type encrypted registered [ 3.712686] ima: No TPM chip found, activating TPM-bypass! [ 3.715268] ima: Allocated hash algorithm: sha1 [ 3.717142] ima: No architecture policies found [ 3.719162] evm: Initialising EVM extended attributes: [ 3.721258] evm: security.selinux [ 3.722494] evm: security.ima [ 3.723723] evm: security.capability [ 3.725206] evm: HMAC attrs: 0x1 [ 3.727825] rtc_cmos 00:05: setting system clock to 2026-09-03 13:05:21 UTC (1788440721) [ 3.734627] debug: unmapping init [mem 0xffffffff9bc03000-0xffffffff9bdfffff] [ 3.738144] debug: unmapping init [mem 0xffffffff9a982000-0xffffffff9ac58fff] [ 3.748158] Write protecting the kernel read-only data: 28672k [ 3.752220] debug: unmapping init [mem 0xffffffff99003000-0xffffffff991fffff] [ 3.755574] debug: unmapping init [mem 0xffffffff99914000-0xffffffff999fffff] [ 3.790906] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.799430] systemd[1]: Detected virtualization kvm. [ 3.801258] systemd[1]: Detected architecture x86-64. [ 3.803271] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.831311] systemd[1]: No hostname configured. [ 3.833112] systemd[1]: Set hostname to . [ 3.835323] random: systemd: uninitialized urandom read (16 bytes read) [ 3.837754] systemd[1]: Initializing machine ID from random generator. [ 3.889815] random: ln: uninitialized urandom read (6 bytes read) [ 3.983323] random: systemd: uninitialized urandom read (16 bytes read) [ 3.985745] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.990062] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.999735] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.601951] device-mapper: uevent: version 1.0.3 [ 4.604656] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.292111] virtio_net virtio0 ens2: renamed from eth0 [ 5.296039] random: fast init done [ 5.404300] scsi host0: ata_piix [ 5.412724] scsi host1: ata_piix [ 5.414418] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.417255] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.998425] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.240118] random: crng init done [ 10.241684] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.607412] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.734179] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.024373] SELinux: Disabled at runtime. [ 12.088233] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.096157] systemd[1]: Detected virtualization kvm. [ 12.097958] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.623691] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.627600] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.632606] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.638315] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.641668] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.649050] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.656688] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.752756] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.112952] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.323262] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.387759] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.511223] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.522713] EDAC sbridge: Ver: 1.1.2 [ 15.183732] Key type dns_resolver registered [ 15.487834] NFS: Registering the id_resolver key type [ 15.489381] Key type id_resolver registered [ 15.490566] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg403-server login: [ 43.258483] libcfs: loading out-of-tree module taints kernel. [ 43.278983] Key type ._llcrypt registered [ 43.280672] Key type .llcrypt registered [ 43.328705] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 62.814309] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 64.730634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.756908] alg: No test for adler32 (adler32-zlib) [ 66.393341] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 67.414134] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 69.250781] Key type lgssc registered [ 71.614684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 90.927954] hrtimer: interrupt took 5770887 ns [ 92.509267] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 139.347922] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 153.700394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 153.735231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.147880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.250757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 155.335119] Lustre: lustre-MDT0000: new disk, initializing [ 155.389786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 155.417994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.978940] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.870739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 175.982771] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 176.013474] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 176.017973] Lustre: Skipped 1 previous similar message [ 176.091162] Lustre: lustre-MDT0001: new disk, initializing [ 176.160102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.182115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 176.194132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 180.992396] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.267261] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.147473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.620832] Lustre: lustre-OST0000: new disk, initializing [ 197.627210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 197.638557] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.708126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.940026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 199.957715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.053204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.833522] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.990323] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.157704] Lustre: lustre-OST0001: new disk, initializing [ 221.161939] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 221.177408] Lustre: 9517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.274857] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.083853] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.960419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 229.970336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.996698] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 239.871914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.546876] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 254.463464] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing check_logdir /tmp/testlogs/ [ 260.314628] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing yml_node [ 267.206263] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 271.468601] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 274.650268] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 276.539742] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:09:51 EDT 2026 [ 297.298655] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 299.064691] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.897201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.906387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.921056] Lustre: Skipped 2 previous similar messages [ 316.953085] Lustre: Skipped 3 previous similar messages [ 322.016672] Lustre: server umount lustre-MDT0000 complete [ 325.922973] LustreError: 8443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441043 with bad export cookie 3909792348737765935 [ 325.928061] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 325.943775] LustreError: 8443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 327.136459] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.137041] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.137450] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 327.173553] Lustre: Skipped 2 previous similar messages [ 327.232789] LustreError: 6511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 330.153702] LustreError: 12150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.154150] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.185707] LustreError: 12150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 330.192854] Lustre: Skipped 2 previous similar messages [ 332.358713] Lustre: server umount lustre-MDT0001 complete [ 336.359599] Lustre: server umount lustre-OST0000 complete [ 339.943877] Lustre: server umount lustre-OST0001 complete [ 346.814522] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 357.567289] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 398.945683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.171847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 399.217260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 399.349465] Lustre: lustre-MDT0000: new disk, initializing [ 399.431841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.451134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 404.415273] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 417.388655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 417.555597] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.649632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.653433] Lustre: Skipped 1 previous similar message [ 417.720755] Lustre: lustre-MDT0001: new disk, initializing [ 417.819563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 417.890317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 417.900657] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 422.536845] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 433.798294] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 440.062252] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.573368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.863716] Lustre: lustre-OST0000: new disk, initializing [ 447.873605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.882540] Lustre: 17043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 448.046500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.568634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.576965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.718559] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 454.861197] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 464.929443] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 470.120977] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 470.328512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.317966] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.522245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.740596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.748426] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.749288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.762685] Lustre: Skipped 1 previous similar message [ 480.782590] LustreError: Skipped 1 previous similar message [ 484.147121] Lustre: server umount lustre-OST0000 complete [ 489.440836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 489.446105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.455539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 489.461305] Lustre: Skipped 1 previous similar message [ 494.497494] Lustre: server umount lustre-MDT0000 complete [ 496.097764] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.310182] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441217 with bad export cookie 3909792348737767475 [ 499.317079] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.326865] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 500.203028] Lustre: server umount lustre-MDT0001 complete [ 505.240766] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:13:40 (1788441220) [ 508.762388] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 511.716937] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:13:46 (1788441226) [ 519.572160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.156697] LustreError: 19453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.172600] LustreError: 19453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 520.205911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 524.900586] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 525.290769] LustreError: 19454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.293254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.092907] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 551.105240] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 559.103660] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 567.233776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 575.455712] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 584.512635] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 605.673777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 605.686256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.700929] Lustre: Skipped 1 previous similar message [ 605.708256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 605.715388] Lustre: Skipped 2 previous similar messages [ 610.754131] Lustre: server umount lustre-OST0000 complete [ 615.919192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 615.939892] LustreError: Skipped 1 previous similar message [ 615.953895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.981989] Lustre: Skipped 1 previous similar message [ 621.996571] Lustre: server umount lustre-MDT0000 complete [ 622.056173] LustreError: 19449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.088799] LustreError: 19449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 626.986731] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441344 with bad export cookie 3909792348737768350 [ 626.998868] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.007161] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 627.358465] Lustre: server umount lustre-MDT0001 complete [ 641.688609] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 645.183953] Key type lgssc unregistered [ 645.596806] LNet: 23018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.615239] LNetError: 23018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.633834] LNet: Removed LNI 192.168.204.103@tcp [ 646.767282] Key type .llcrypt unregistered [ 646.774518] Key type ._llcrypt unregistered [ 673.455640] Key type ._llcrypt registered [ 673.461802] Key type .llcrypt registered [ 673.686271] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 674.714771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 674.806035] alg: No test for adler32 (adler32-zlib) [ 676.055754] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 676.414097] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 678.111452] Key type lgssc registered [ 679.311995] Lustre: Echo OBD driver; http://www.lustre.org/ [ 689.816596] loop: module loaded [ 689.976622] loop0: detected capacity change from 0 to 204800000 [ 690.266107] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.285841] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.712243] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.452138] loop0: detected capacity change from 0 to 204800000 [ 697.691993] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.703154] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.717765] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 704.156211] loop0: detected capacity change from 0 to 204800000 [ 710.241958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 710.266413] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.476406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 711.552347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 711.644258] Lustre: lustre-MDT0000: new disk, initializing [ 711.734991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.754733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 717.904522] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 728.671058] loop1: detected capacity change from 0 to 204800000 [ 735.413987] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 736.033900] Lustre: lustre-OST07c6: new disk, initializing [ 736.037398] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 736.048504] Lustre: Skipped 1 previous similar message [ 736.051054] Lustre: 26209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 736.149334] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 743.072780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 743.089702] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 743.132829] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 744.236242] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 750.443072] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 750.449592] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 750.454151] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 750.459939] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 750.465708] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 750.470539] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 771.964812] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 771.982601] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 771.996247] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.006944] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 772.020502] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 772.031516] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.419794] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 777.450974] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 777.463492] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 777.476075] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.490398] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.511886] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.525668] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 777.534881] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.545523] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 777.549220] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.558887] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.564770] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 782.955516] Lustre: 26217:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 782.969156] Lustre: 26217:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 782.984736] Lustre: 26217:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 783.003861] Lustre: 26217:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.021779] Lustre: 26217:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 783.034831] Lustre: 26217:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.045693] Lustre: 26217:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 783.057908] Lustre: 26217:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.069116] Lustre: 26217:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 783.084794] Lustre: 26217:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 783.090146] Lustre: 26217:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 783.098849] Lustre: 26217:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.670618] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 789.684270] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 789.692788] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 789.708883] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.719458] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.741912] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.755411] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 789.767693] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.779406] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 789.786210] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 789.794666] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.816729] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 798.238519] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 798.261755] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 798.271341] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 798.286248] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 798.296729] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 798.313972] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 798.329694] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 798.343646] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 798.358905] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 798.366053] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 798.375350] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 798.382444] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 850.420384] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.443487] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 854.339529] Lustre: server umount lustre-OST07c6 complete [ 865.139523] Lustre: server umount lustre-MDT0000 complete [ 885.661306] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 889.255395] Key type lgssc unregistered [ 889.608574] LNet: 28771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.620958] LNetError: 28771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.639973] LNet: Removed LNI 192.168.204.103@tcp [ 890.600401] Key type .llcrypt unregistered [ 890.602168] Key type ._llcrypt unregistered [ 910.831749] Key type ._llcrypt registered [ 910.835264] Key type .llcrypt registered [ 910.942854] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 912.045704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.185577] alg: No test for adler32 (adler32-zlib) [ 913.259204] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 913.621581] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 915.343172] Key type lgssc registered [ 916.556361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.895289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 924.925098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.465931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.735525] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 936.932046] LustreError: 30045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 936.944569] LustreError: 30045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 941.342813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.833260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 946.878881] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 956.097361] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 962.985504] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 970.266126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 970.724984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 971.762689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 977.874460] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 985.738743] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1004.103616] Lustre: server umount lustre-OST0000 complete [ 1008.609162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1008.618209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.632223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1012.706751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.708721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1012.742212] Lustre: Skipped 1 previous similar message [ 1017.825794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.844902] Lustre: Skipped 1 previous similar message [ 1018.959704] Lustre: server umount lustre-MDT0000 complete [ 1022.945748] LustreError: 30045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1022.972035] LustreError: 30045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1023.608319] LustreError: 30025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441741 with bad export cookie 7197932382174597627 [ 1023.641793] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1034.386346] Lustre: server umount lustre-MDT0001 complete [ 1048.729818] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 1052.950109] Key type lgssc unregistered [ 1053.296758] LNet: 33787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.301710] LNetError: 33787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.376786] LNet: Removed LNI 192.168.204.103@tcp [ 1055.551882] Key type .llcrypt unregistered [ 1055.554345] Key type ._llcrypt unregistered [ 1075.843821] Key type ._llcrypt registered [ 1075.850627] Key type .llcrypt registered [ 1075.961280] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1076.976754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1077.108055] alg: No test for adler32 (adler32-zlib) [ 1078.167644] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1078.417826] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 1080.151186] Key type lgssc registered [ 1081.364777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1089.214706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1089.270571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.738754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.075214] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1100.263248] LustreError: 35057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1100.289544] LustreError: 35057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1104.602197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1105.094864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1109.807755] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1118.372727] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1125.217271] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1132.037233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1132.397793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1135.483377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1138.748808] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1147.405985] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1160.166286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.171198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1160.200637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1160.211540] Lustre: Skipped 1 previous similar message [ 1164.474675] Lustre: server umount lustre-OST0000 complete [ 1170.400805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.442534] Lustre: Skipped 1 previous similar message [ 1170.460225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1170.475489] Lustre: Skipped 1 previous similar message [ 1175.198458] Lustre: server umount lustre-MDT0000 complete [ 1175.523240] LustreError: 35786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.537349] LustreError: 35786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1179.139796] LustreError: 35039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441896 with bad export cookie 18019961058732251267 [ 1179.144725] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1179.160220] LustreError: 35039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1179.528352] Lustre: server umount lustre-MDT0001 complete [ 1192.070924] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 1195.257372] Key type lgssc unregistered [ 1195.631726] LNet: 38596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.645241] LNetError: 38596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.681656] LNet: Removed LNI 192.168.204.103@tcp [ 1196.874177] Key type .llcrypt unregistered [ 1196.878569] Key type ._llcrypt unregistered [ 1225.355853] Key type ._llcrypt registered [ 1225.359968] Key type .llcrypt registered [ 1225.506289] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1226.714717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1226.766440] alg: No test for adler32 (adler32-zlib) [ 1227.791933] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1228.053335] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 1229.727211] Key type lgssc registered [ 1230.875525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1240.478951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1240.501571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.923534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.519959] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1251.371500] LustreError: 40015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.385278] LustreError: 40015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1256.423684] LustreError: 40016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.863080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.369879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1263.313883] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1272.358908] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1278.936374] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1285.739537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1286.152333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.165698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1292.415825] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1300.206832] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1313.101589] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.3@tcp (stopping) [ 1313.775191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1313.793914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1313.796556] Lustre: Skipped 2 previous similar messages [ 1313.810157] Lustre: Skipped 2 previous similar messages [ 1316.421473] Lustre: server umount lustre-MDT0000 complete [ 1318.175237] LustreError: 40011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.871033] LustreError: 39997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442037 with bad export cookie 12352935431706201334 [ 1319.878929] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1319.887048] LustreError: 39997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1323.295083] LustreError: 40010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1323.327707] LustreError: 40010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1323.345429] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.3@tcp (stopping) [ 1324.001265] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1328.414212] LustreError: 41488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1328.415513] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.3@tcp (stopping) [ 1328.458689] LustreError: 41488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1328.469092] Lustre: Skipped 2 previous similar messages [ 1333.534644] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.3@tcp (stopping) [ 1333.557698] Lustre: Skipped 1 previous similar message [ 1334.239544] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1334.459168] Lustre: server umount lustre-MDT0001 complete [ 1349.934848] Lustre: server umount lustre-OST0000 complete [ 1368.304734] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 1371.326197] Key type lgssc unregistered [ 1371.561025] LNet: 43885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1371.568542] LNetError: 43885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.580909] LNet: Removed LNI 192.168.204.103@tcp [ 1372.397762] Key type .llcrypt unregistered [ 1372.400278] Key type ._llcrypt unregistered [ 1398.255514] Key type ._llcrypt registered [ 1398.263868] Key type .llcrypt registered [ 1398.391928] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1399.546362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1399.634134] alg: No test for adler32 (adler32-zlib) [ 1400.766800] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1401.138663] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 1402.863318] Key type lgssc registered [ 1404.497821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.669939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.705516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.289820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.269744] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1425.894393] LustreError: 45302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.913490] LustreError: 45302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1430.696222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.158769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1436.036768] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1444.674940] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1451.700610] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1459.814149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1460.361230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.429265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1467.790378] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1476.670459] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1484.620693] Lustre: lustre-OST0000: Not available for connect from 192.168.204.3@tcp (stopping) [ 1486.306773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1486.307230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.324472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.353582] Lustre: Skipped 1 previous similar message [ 1489.707307] LustreError: 47124:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.899265] Lustre: server umount lustre-OST0000 complete [ 1511.913287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.921041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1511.925868] Lustre: Skipped 1 previous similar message [ 1511.934555] Lustre: Skipped 1 previous similar message [ 1516.682342] Lustre: server umount lustre-MDT0000 complete [ 1517.026954] LustreError: 46031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1521.172552] LustreError: 45283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442238 with bad export cookie 17708015919552349173 [ 1521.179528] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1521.189515] LustreError: 45283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1527.801903] Lustre: server umount lustre-MDT0001 complete [ 1540.186677] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 1543.216178] Key type lgssc unregistered [ 1543.588612] LNet: 49014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.599074] LNetError: 49014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.631066] LNet: Removed LNI 192.168.204.103@tcp [ 1544.448622] Key type .llcrypt unregistered [ 1544.452942] Key type ._llcrypt unregistered [ 1571.081981] Key type ._llcrypt registered [ 1571.088328] Key type .llcrypt registered [ 1571.303831] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1573.157991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1573.408848] alg: No test for adler32 (adler32-zlib) [ 1574.611509] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1575.013287] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 1576.807147] Key type lgssc registered [ 1578.299805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1587.650215] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1587.690573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.282528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1595.401499] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1598.435450] LustreError: 50430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.449450] LustreError: 50430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1603.557403] LustreError: 50431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.355500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.878603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1611.188587] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1620.373428] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1627.101857] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1634.529447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1635.139320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1643.088766] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1651.287250] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1658.689754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1666.589041] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1671.145733] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1684.054304] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 9h33m18s [ 1686.209240] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1689.571377] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.590685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1689.598650] Lustre: Skipped 1 previous similar message [ 1689.615270] Lustre: Skipped 2 previous similar messages [ 1690.664545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1690.680397] Lustre: Skipped 2 previous similar messages [ 1692.986710] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.3@tcp (stopping) [ 1695.152514] Lustre: server umount lustre-MDT0000 complete [ 1696.753513] LustreError: 52923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1696.767024] LustreError: 52923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1699.679857] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1700.132470] LustreError: 52215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.263905] LustreError: 52215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.299588] LustreError: 52215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1707.924522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.083114] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1708.352919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1708.482152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1709.364258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1713.366541] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1717.420922] Lustre: DEBUG MARKER: Wait for df (36756) ... [ 1719.454027] Lustre: DEBUG MARKER: done [ 1724.897334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.906480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1724.929302] Lustre: Skipped 2 previous similar messages [ 1724.942545] Lustre: Skipped 1 previous similar message [ 1730.023849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1730.030313] Lustre: Skipped 1 previous similar message [ 1739.231166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1739.431154] Lustre: server umount lustre-OST0000 complete [ 1745.376745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1745.393062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1745.396473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1745.396481] Lustre: Skipped 3 previous similar messages [ 1745.437062] Lustre: Skipped 1 previous similar message [ 1749.760607] Lustre: server umount lustre-MDT0000 complete [ 1752.551804] LustreError: 50430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1752.572233] LustreError: 50430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1753.895758] LustreError: 53083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442471 with bad export cookie 5974428637595718937 [ 1753.906568] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1753.911241] LustreError: 53083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1754.404777] Lustre: server umount lustre-MDT0001 complete [ 1767.279845] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 1770.185830] Key type lgssc unregistered [ 1770.632232] LNet: 55390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.655872] LNetError: 55390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1770.697477] LNet: Removed LNI 192.168.204.103@tcp [ 1771.820384] Key type .llcrypt unregistered [ 1771.823840] Key type ._llcrypt unregistered [ 1813.487981] Key type ._llcrypt registered [ 1813.494665] Key type .llcrypt registered [ 1813.603413] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 1815.117396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1815.162598] alg: No test for adler32 (adler32-zlib) [ 1816.221569] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1816.609991] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 1818.271179] Key type lgssc registered [ 1819.641231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1828.982044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.026269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.406714] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1830.468350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1830.687642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1835.768444] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1845.566092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.714996] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1845.765600] Lustre: 57598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1845.989212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1851.405596] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1859.667964] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1866.509885] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1873.900719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.124536] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1874.376577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.227379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1881.557781] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 1890.150776] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1897.528862] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h36m52s [ 1899.241301] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1907.138887] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1909.770081] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1909.949439] LustreError: 57605:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1930.064628] Lustre: lustre-MDT0000: Client 7b2e657c-d743-424a-bfca-12954e810db6 (at 192.168.204.3@tcp) reconnecting [ 1949.951299] LustreError: 57605:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1952.508452] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1955.095193] Lustre: DEBUG MARKER: done [ 1962.982541] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1962.988791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1963.008286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1963.491209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1963.516073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1963.521320] Lustre: Skipped 1 previous similar message [ 1968.615265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.732709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1973.745606] Lustre: Skipped 2 previous similar messages [ 1975.263148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1975.599290] Lustre: server umount lustre-OST0000 complete [ 1983.989813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.008774] Lustre: Skipped 1 previous similar message [ 1984.009341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1984.032948] Lustre: Skipped 1 previous similar message [ 1986.553516] Lustre: server umount lustre-MDT0000 complete [ 1991.098741] LustreError: 57591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442708 with bad export cookie 12883194099146462280 [ 1991.113498] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1991.122887] LustreError: 57591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1991.589935] Lustre: server umount lustre-MDT0001 complete [ 2005.062741] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 2008.595393] Key type lgssc unregistered [ 2009.060497] LNet: 61931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.074892] LNetError: 61931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.091597] LNet: Removed LNI 192.168.204.103@tcp [ 2010.156805] Key type .llcrypt unregistered [ 2010.158778] Key type ._llcrypt unregistered [ 2058.640886] Key type ._llcrypt registered [ 2058.646283] Key type .llcrypt registered [ 2058.812933] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 2059.974039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2060.119506] alg: No test for adler32 (adler32-zlib) [ 2061.438871] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2061.874386] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 2063.607220] Key type lgssc registered [ 2064.996439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2069.889309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2069.910501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2071.425898] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2071.450258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2071.736736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.874502] Lustre: Failing over lustre-MDT0000 [ 2073.485701] Lustre: server umount lustre-MDT0000 complete [ 2081.903311] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:39:57 (1788442797) [ 2089.908766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.394936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.306674] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2106.494510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.587915] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2106.616488] Lustre: 64741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2106.915566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2112.310452] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2121.603186] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2128.492866] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2136.373290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2136.612815] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2136.833886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2143.444938] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2146.498541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2152.454878] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2158.528589] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2167.273752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2167.287503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2167.297125] Lustre: Skipped 2 previous similar messages [ 2167.308175] Lustre: Skipped 2 previous similar messages [ 2172.386233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2172.394337] Lustre: Skipped 2 previous similar messages [ 2172.720361] Lustre: server umount lustre-MDT0000 complete [ 2177.022668] LustreError: 64732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442894 with bad export cookie 8616213201670467558 [ 2177.030832] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.031026] LustreError: 64732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2177.505060] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2177.618432] Lustre: server umount lustre-MDT0001 complete [ 2181.736468] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2187.387443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2196.557860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.080984] LustreError: 63657:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ad68193d x1875318175475328/t0(0) o250->MGC192.168.204.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2202.634746] LustreError: 66590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2202.665403] LustreError: 66590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2202.760792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.896686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2207.910736] LustreError: 66591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2207.913148] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2208.507042] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2213.347426] LustreError: 66590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.466839] LustreError: 68513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.524681] LustreError: 68513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2220.531367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.032270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2226.172177] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2226.549409] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2236.102140] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2243.046881] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2251.760483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.762728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2251.763562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2251.773836] Lustre: Skipped 2 previous similar messages [ 2251.781896] Lustre: Skipped 1 previous similar message [ 2256.870489] LustreError: 68513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2256.897642] LustreError: 68513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2257.039740] Lustre: server umount lustre-MDT0000 complete [ 2261.741807] LustreError: 64732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442979 with bad export cookie 8616213201670469833 [ 2261.764349] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2261.995746] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2262.018635] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2262.024440] Lustre: Skipped 1 previous similar message [ 2262.312266] Lustre: server umount lustre-MDT0001 complete [ 2269.529370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2280.507818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.561627] LustreError: 63657:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008a3fc572 x1875318175503488/t0(0) o250->MGC192.168.204.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2286.879388] LustreError: 66590:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.904783] LustreError: 66590:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2286.968207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.088091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2292.080159] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2293.079358] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2303.302670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.709843] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2309.096719] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2316.892989] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2324.023936] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2334.697070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.715061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.724872] Lustre: Skipped 2 previous similar messages [ 2334.726943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2334.734694] Lustre: Skipped 2 previous similar messages [ 2338.776981] Lustre: server umount lustre-MDT0000 complete [ 2339.824050] LustreError: 71197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.841724] LustreError: 71197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2343.504346] LustreError: 64731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443061 with bad export cookie 8616213201670471212 [ 2343.524600] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.528144] LustreError: 64731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2344.001412] Lustre: server umount lustre-MDT0001 complete [ 2351.750836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2356.401388] Lustre: DEBUG MARKER: files compared the same [ 2368.655676] Lustre: server umount lustre-OST0000 complete [ 2389.878631] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 2393.768444] Key type lgssc unregistered [ 2394.149664] LNet: 74480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2394.158000] LNetError: 74480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.175892] LNet: Removed LNI 192.168.204.103@tcp [ 2395.266808] Key type .llcrypt unregistered [ 2395.274100] Key type ._llcrypt unregistered [ 2423.172835] Key type ._llcrypt registered [ 2423.177034] Key type .llcrypt registered [ 2423.417343] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 2425.040096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2425.075097] alg: No test for adler32 (adler32-zlib) [ 2426.426950] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2426.876356] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 2428.591915] Key type lgssc registered [ 2430.256112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2440.181853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2440.220993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2441.941867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2448.644139] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug +malloc all [ 2451.426030] LustreError: 75898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.452741] LustreError: 75898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2456.545184] LustreError: 75899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.738521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.030073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2466.841101] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug +malloc all [ 2476.285904] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2484.126124] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2492.788376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2493.314265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.038412] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug +malloc all [ 2510.374991] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2512.757536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2526.176123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2526.194462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.228506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2528.224177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2528.228066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2528.242716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.509609] Lustre: server umount lustre-OST0000 complete [ 2537.952305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2537.957636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.991937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2537.998862] Lustre: Skipped 1 previous similar message [ 2543.195101] Lustre: server umount lustre-MDT0000 complete [ 2543.586511] LustreError: 76627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.600756] LustreError: 76627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2547.472507] LustreError: 75880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443265 with bad export cookie 2506128855469353697 [ 2547.482540] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2547.751985] Lustre: server umount lustre-MDT0001 complete [ 2561.326820] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 2564.643831] Key type lgssc unregistered [ 2564.983754] LNet: 79458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2564.988596] LNetError: 79458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2565.003464] LNet: Removed LNI 192.168.204.103@tcp [ 2566.041053] Key type .llcrypt unregistered [ 2566.043076] Key type ._llcrypt unregistered [ 2594.906343] Key type ._llcrypt registered [ 2594.908828] Key type .llcrypt registered [ 2594.996296] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 2596.046045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2596.081334] alg: No test for adler32 (adler32-zlib) [ 2597.180482] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2597.457265] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 2599.127267] Key type lgssc registered [ 2600.284678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2612.751206] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2612.771088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2628.320873] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2633.441292] LustreError: 80850:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.103@tcp: failed processing log, type 4: rc = -110 [ 2660.127970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.533700] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2675.626869] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2684.520261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.717522] Lustre: *** cfs_fail_loc=706, val=0*** [ 2685.294311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.382802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2690.533694] LustreError: 80876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.566326] LustreError: 80876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2690.867298] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2694.626207] LustreError: 81938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.676051] LustreError: 81938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2699.751575] LustreError: 81936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2699.783127] LustreError: 81936:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2700.188389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.434335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2705.159926] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2715.788611] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2723.864579] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2731.495232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2731.498558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2731.528140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2733.227497] Lustre: server umount lustre-OST0000 complete [ 2741.217360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2741.225350] LustreError: Skipped 1 previous similar message [ 2741.232025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.249321] Lustre: Skipped 1 previous similar message [ 2741.253262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2741.262512] Lustre: Skipped 1 previous similar message [ 2744.421770] Lustre: server umount lustre-MDT0000 complete [ 2746.851742] LustreError: 81941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.876904] LustreError: 81941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2748.499278] LustreError: 80856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443466 with bad export cookie 16896709690905734088 [ 2748.508662] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2748.513886] LustreError: 80856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2748.829566] Lustre: server umount lustre-MDT0001 complete [ 2760.847574] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 2763.996508] Key type lgssc unregistered [ 2764.377331] LNet: 84473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2764.386375] LNetError: 84473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2764.403406] LNet: Removed LNI 192.168.204.103@tcp [ 2765.438234] Key type .llcrypt unregistered [ 2765.440587] Key type ._llcrypt unregistered [ 2793.087646] Key type ._llcrypt registered [ 2793.092243] Key type .llcrypt registered [ 2793.261802] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 2794.670692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2794.685317] alg: No test for adler32 (adler32-zlib) [ 2795.783779] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2796.095224] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 2797.840816] Key type lgssc registered [ 2799.160381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2807.903123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2807.925377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.095863] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2823.384208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2824.031668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2829.284743] LustreError: 86577:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.314393] LustreError: 86577:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2829.403851] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2833.380021] LustreError: 86578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.401524] LustreError: 86578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2838.504658] LustreError: 86582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.535400] LustreError: 86582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2839.781641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2840.042730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.157458] LustreError: 86583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2845.180779] LustreError: 86583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2847.216607] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2849.267404] LustreError: 86582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2849.298190] LustreError: 86582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2856.778942] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2856.798483] Lustre: MGS: Client e2709ff4-c630-417e-8d01-4b8eb7b647fb (at 0@lo) reconnecting [ 2856.814626] LustreError: 86593:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2857.072169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.156977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2861.964560] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 2877.920991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2877.927299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.966613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2882.529293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2882.542420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.554726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2883.917270] Lustre: server umount lustre-OST0000 complete [ 2893.285451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2893.294680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.296092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2893.296099] Lustre: Skipped 2 previous similar messages [ 2893.328338] Lustre: Skipped 1 previous similar message [ 2894.757349] Lustre: server umount lustre-MDT0000 complete [ 2898.400958] LustreError: 88123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.432234] LustreError: 88123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2899.495905] Lustre: server umount lustre-MDT0001 complete [ 2903.683907] Lustre: server umount MGS complete [ 2916.210250] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:53:51 (1788443631) [ 2945.420683] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 2956.154276] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3000.182729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3005.072851] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3014.842264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3015.134944] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3015.222082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3015.358151] Lustre: lustre-MDT0001: new disk, initializing [ 3015.463090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3015.502106] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3015.514993] Lustre: 93771:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3015.530238] LustreError: 93771:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3020.576457] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3032.147189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3032.599616] Lustre: lustre-OST0000: new disk, initializing [ 3032.610576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3032.619541] Lustre: 94538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3032.732966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.865081] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3042.408042] Lustre: 94994:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000016482414 x1875318945130112/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788443776 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3048.110311] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3055.619375] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3055.635798] Lustre: MGS: Client d40c6019-f9f5-4af6-9c53-61256a83d690 (at 0@lo) reconnecting [ 3055.718127] Lustre: 93038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3055.746329] LustreError: 93757:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3055.791738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3055.969384] Lustre: lustre-MDT0000: new disk, initializing [ 3056.190230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.239524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3056.409817] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3057.314847] Lustre: 95607:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b26177c2 x1875318945136768/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788443791 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3061.025800] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3062.751854] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3066.425861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3066.445128] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3066.652370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3092.452667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3092.460671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.474267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3092.477680] Lustre: Skipped 1 previous similar message [ 3100.128929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3100.157262] Lustre: Skipped 3 previous similar messages [ 3106.271532] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3106.584597] Lustre: server umount lustre-OST0000 complete [ 3115.488751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3115.500987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.511203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3115.519785] Lustre: Skipped 2 previous similar messages [ 3115.532191] Lustre: Skipped 3 previous similar messages [ 3117.646820] Lustre: server umount lustre-MDT0000 complete [ 3120.609505] LustreError: 93743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3120.641060] LustreError: 93743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3128.568104] Lustre: server umount lustre-MDT0001 complete [ 3132.882804] Lustre: server umount MGS complete [ 3146.793649] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:57:42 (1788443862) [ 3166.953181] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 3171.323649] Key type lgssc unregistered [ 3171.711894] LNet: 98109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3171.719905] LNetError: 98109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3171.771801] LNet: Removed LNI 192.168.204.103@tcp [ 3173.246785] Key type .llcrypt unregistered [ 3173.255853] Key type ._llcrypt unregistered [ 3196.976222] Key type ._llcrypt registered [ 3196.979379] Key type .llcrypt registered [ 3197.094424] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3198.594223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3198.634242] alg: No test for adler32 (adler32-zlib) [ 3199.777823] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3200.015540] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 3201.703277] Key type lgssc registered [ 3203.055286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3212.563230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3212.577030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.149337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.535787] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3223.534055] LustreError: 99335:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.550534] LustreError: 99335:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3228.652032] LustreError: 99334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.202099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3230.602851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3235.624301] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3244.915651] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3252.248189] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3260.737233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3261.369375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3269.045668] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3278.747701] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3283.880742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3294.202441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.204460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3294.235595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.255329] Lustre: Skipped 1 previous similar message [ 3299.311489] LustreError: 101159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3299.544931] Lustre: server umount lustre-OST0000 complete [ 3304.421144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.426437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.444293] Lustre: Skipped 1 previous similar message [ 3304.462977] Lustre: Skipped 2 previous similar messages [ 3309.537759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.554912] Lustre: Skipped 1 previous similar message [ 3310.548193] Lustre: server umount lustre-MDT0000 complete [ 3314.125316] LustreError: 101179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444031 with bad export cookie 10211919432416401103 [ 3314.137421] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3314.142086] LustreError: 101179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3314.543429] Lustre: server umount lustre-MDT0001 complete [ 3327.206803] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 3330.550103] Key type lgssc unregistered [ 3330.890926] LNet: 102892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3330.899115] LNetError: 102892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3330.924454] LNet: Removed LNI 192.168.204.103@tcp [ 3331.950299] Key type .llcrypt unregistered [ 3331.952721] Key type ._llcrypt unregistered [ 3349.444166] Key type ._llcrypt registered [ 3349.446314] Key type .llcrypt registered [ 3349.549795] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3350.279356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3350.322416] alg: No test for adler32 (adler32-zlib) [ 3351.312748] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3351.622132] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 3353.431171] Key type lgssc registered [ 3354.938868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3364.403363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3364.412481] LustreError: 103982:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3364.492715] LustreError: 103992:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3364.500394] LustreError: 103982:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4926 [ 3364.515713] LustreError: 103982:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3364.522369] LustreError: 103982:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3364.533712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.485327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.888274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.174031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.755964] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3386.496433] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing lsmod [ 3389.391225] LustreError: 105193:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3389.443587] LustreError: 105200:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3389.450051] LustreError: 105193:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4949 [ 3389.457719] LustreError: 105193:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3389.470602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.476622] LustreError: 105193:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3389.777629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.870947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3396.584465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3396.587944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.618811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3397.090125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.850822] Lustre: server umount lustre-MDT0000 complete [ 3401.249753] LustreError: 104030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.280914] LustreError: 104030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3404.991898] LustreError: 104005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444122 with bad export cookie 6092552533624943932 [ 3405.014104] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3405.432544] LustreError: 104461:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002b04f6bb x1875319528117376/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3405.737460] Lustre: server umount lustre-MDT0001 complete [ 3421.013546] Lustre: server umount lustre-OST0000 complete [ 3429.537467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.895052] LustreError: 106262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.965962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3434.713735] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3434.977147] LustreError: 106263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3440.103969] LustreError: 106262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3444.418660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3444.952934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3449.562324] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3457.534108] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3463.808327] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3471.531507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3472.143802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.204575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3479.360902] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3489.091650] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3495.907659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3495.915973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3495.932352] Lustre: Skipped 1 previous similar message [ 3495.937295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3495.944707] Lustre: Skipped 3 previous similar messages [ 3500.005766] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3500.009163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3500.022050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.866039] Lustre: server umount lustre-OST0000 complete [ 3506.657343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3506.671910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3506.696057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3506.709445] Lustre: Skipped 1 previous similar message [ 3511.779971] Lustre: server umount lustre-MDT0000 complete [ 3515.368562] LustreError: 106259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3515.388212] LustreError: 106259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3515.741754] LustreError: 106245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444233 with bad export cookie 6092552533624944863 [ 3515.764495] LustreError: 106245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3515.773736] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3516.080027] Lustre: server umount lustre-MDT0001 complete [ 3527.904705] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 3531.647797] Key type lgssc unregistered [ 3532.160437] LNet: 109801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3532.175293] LNetError: 109801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3532.201903] LNet: Removed LNI 192.168.204.103@tcp [ 3533.079768] Key type .llcrypt unregistered [ 3533.082514] Key type ._llcrypt unregistered [ 3561.734905] Key type ._llcrypt registered [ 3561.739635] Key type .llcrypt registered [ 3561.864332] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3563.219854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3563.442292] alg: No test for adler32 (adler32-zlib) [ 3564.572449] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3564.926701] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 3566.667399] Key type lgssc registered [ 3568.177830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3577.474265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3577.489798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.941469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.412167] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3589.102900] LustreError: 111220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3589.134836] LustreError: 111220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3594.211950] LustreError: 111219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3595.438065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.966468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3602.293407] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3611.740145] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3619.195207] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3628.074880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.636187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3635.495316] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3644.737664] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3649.154534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3657.617210] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3662.756978] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3669.987739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3669.996432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3670.010444] Lustre: Skipped 1 previous similar message [ 3670.012100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3670.053961] Lustre: Skipped 1 previous similar message [ 3673.641897] Lustre: server umount lustre-OST0000 complete [ 3680.225469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3680.229816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3680.254920] Lustre: Skipped 1 previous similar message [ 3684.731980] Lustre: server umount lustre-MDT0000 complete [ 3685.352722] LustreError: 112317:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.383531] LustreError: 112317:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3689.449445] LustreError: 111214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3689.483417] LustreError: 111214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3690.078780] LustreError: 113779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444407 with bad export cookie 1993194719561670747 [ 3690.085378] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3690.094518] LustreError: 113779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3690.449688] Lustre: server umount lustre-MDT0001 complete [ 3703.367168] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 3707.078282] Key type lgssc unregistered [ 3707.613445] LNet: 114890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3707.626545] LNetError: 114890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3707.648259] LNet: Removed LNI 192.168.204.103@tcp [ 3708.900349] Key type .llcrypt unregistered [ 3708.907139] Key type ._llcrypt unregistered [ 3727.958055] Key type ._llcrypt registered [ 3727.960429] Key type .llcrypt registered [ 3728.051376] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3728.997092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3729.141603] alg: No test for adler32 (adler32-zlib) [ 3730.556081] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3730.856963] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 3732.607191] Key type lgssc registered [ 3734.422766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3743.527769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3743.542917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3744.998586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.405767] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3754.467802] LustreError: 116113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3754.501913] LustreError: 116113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3758.522417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.275364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3764.129186] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3774.523844] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3781.622461] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3789.583426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3790.020866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.108699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3795.766875] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3805.157152] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3816.929349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3816.942923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3816.961653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3818.983617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3818.991383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.995821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3820.670740] Lustre: server umount lustre-OST0000 complete [ 3825.632465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3825.643452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3825.660280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3825.668791] Lustre: Skipped 1 previous similar message [ 3829.217266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3829.223838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.632210] Lustre: server umount lustre-MDT0000 complete [ 3834.342957] LustreError: 117127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.387660] LustreError: 117127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3836.523656] LustreError: 116093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444554 with bad export cookie 4985184787907993361 [ 3836.526700] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.532325] LustreError: 116093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3837.275362] Lustre: server umount lustre-MDT0001 complete [ 3851.200820] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 3854.828848] Key type lgssc unregistered [ 3855.142875] LNet: 119649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3855.157304] LNetError: 119649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3855.167563] LNet: Removed LNI 192.168.204.103@tcp [ 3856.160499] Key type .llcrypt unregistered [ 3856.163099] Key type ._llcrypt unregistered [ 3884.781917] Key type ._llcrypt registered [ 3884.790356] Key type .llcrypt registered [ 3884.897512] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 3886.335064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3886.526606] alg: No test for adler32 (adler32-zlib) [ 3887.579213] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3887.779687] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 3889.451610] Key type lgssc registered [ 3890.926597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3899.404789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3899.432929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3900.988090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.811578] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3911.141476] LustreError: 121066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3911.180723] LustreError: 121066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3916.119107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.258486] LustreError: 121067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3916.608332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3922.143899] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3932.262957] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3938.515784] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3945.664674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3945.965762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3953.176285] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 3961.852272] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3970.610566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3978.650944] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3982.722467] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3982.732691] Lustre: Skipped 2 previous similar messages [ 3986.570095] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3986.635972] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3987.969870] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3987.978203] Lustre: Skipped 2 previous similar messages [ 3995.697700] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3995.701453] Lustre: Skipped 1 previous similar message [ 4002.224293] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4002.238110] Lustre: Skipped 2 previous similar messages [ 4012.633373] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.3@tcp 192.168.0.[2,10]@tcp [ 4016.266243] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.3/24@tcp in log lustre-MDT0000 [ 4016.270607] Lustre: Skipped 2 previous similar messages [ 4022.406222] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.3/24@tcp [ 4022.409792] Lustre: Skipped 1 previous similar message [ 4031.973680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4031.977444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4032.015834] Lustre: Skipped 1 previous similar message [ 4032.019207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4032.022021] Lustre: Skipped 1 previous similar message [ 4037.096640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4037.107783] Lustre: Skipped 1 previous similar message [ 4037.467579] Lustre: server umount lustre-OST0000 complete [ 4042.208636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4042.211075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4042.231572] Lustre: Skipped 1 previous similar message [ 4042.253348] Lustre: Skipped 1 previous similar message [ 4047.329731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4047.344375] Lustre: Skipped 1 previous similar message [ 4048.382890] Lustre: server umount lustre-MDT0000 complete [ 4052.228930] LustreError: 121048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444769 with bad export cookie 10580725064164708003 [ 4052.231170] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4052.246950] LustreError: 121048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4052.783334] Lustre: server umount lustre-MDT0001 complete [ 4066.675594] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 4070.472526] Key type lgssc unregistered [ 4070.871786] LNet: 126234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4070.879362] LNetError: 126234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4070.923154] LNet: Removed LNI 192.168.204.103@tcp [ 4072.031218] Key type .llcrypt unregistered [ 4072.037531] Key type ._llcrypt unregistered [ 4102.838290] Key type ._llcrypt registered [ 4102.851795] Key type .llcrypt registered [ 4103.074453] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 4105.800355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4105.875154] alg: No test for adler32 (adler32-zlib) [ 4107.003235] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4107.295453] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 4109.039853] Key type lgssc registered [ 4110.756816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4120.319121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4120.368615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.164190] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4122.184633] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.3/24@tcp [ 4122.212231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4128.624828] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4131.305774] LustreError: 127796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4131.388448] LustreError: 127796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4136.417782] LustreError: 127797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4140.149416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.419099] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4140.465418] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.3/24@tcp [ 4140.498898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4145.079034] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4153.358470] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4160.513378] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4168.673103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4169.244956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4176.293106] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4185.292205] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4191.759241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4206.050316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4206.062410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.088129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4207.073043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4207.078959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4207.086261] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4207.126390] Lustre: Skipped 1 previous similar message [ 4212.109479] Lustre: server umount lustre-OST0000 complete [ 4218.520344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4218.528736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4218.556160] Lustre: Skipped 1 previous similar message [ 4218.584669] Lustre: Skipped 1 previous similar message [ 4223.968422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4223.984054] Lustre: Skipped 1 previous similar message [ 4225.029097] Lustre: server umount lustre-MDT0000 complete [ 4229.110568] LustreError: 128815:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4229.143698] LustreError: 128815:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4230.318552] LustreError: 127778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444948 with bad export cookie 12273852869383014396 [ 4230.335782] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4230.829404] Lustre: server umount lustre-MDT0001 complete [ 4244.654363] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 4248.747232] Key type lgssc unregistered [ 4249.286738] LNet: 131423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4249.305371] LNetError: 131423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4249.364314] LNet: Removed LNI 192.168.204.103@tcp [ 4250.923752] Key type .llcrypt unregistered [ 4250.926465] Key type ._llcrypt unregistered [ 4284.943618] Key type ._llcrypt registered [ 4284.946871] Key type .llcrypt registered [ 4285.069624] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 4304.833398] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 4306.699145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4306.909083] alg: No test for adler32 (adler32-zlib) [ 4307.993330] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4308.191951] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 4309.978972] Key type lgssc registered [ 4311.520968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4353.651920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4353.678364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4354.997902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4355.045831] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4355.180669] Lustre: lustre-MDT0000: new disk, initializing [ 4355.362176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4355.397205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4360.957854] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4373.851970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.058372] Lustre: 134824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4374.113508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4374.118082] Lustre: Skipped 1 previous similar message [ 4374.215468] Lustre: lustre-MDT0001: new disk, initializing [ 4374.359624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4374.422921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4374.435360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4379.690830] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4391.096654] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4398.187273] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4406.324762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4406.591946] Lustre: lustre-OST0000: new disk, initializing [ 4406.595771] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4406.602241] Lustre: 136793:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4406.707707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.982646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4411.999964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4412.111914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4414.765459] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4425.724589] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4431.686761] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4431.919432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4437.498255] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4437.772564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4443.106357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4443.112874] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.116780] LustreError: Skipped 1 previous similar message [ 4443.118347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4443.154260] Lustre: Skipped 1 previous similar message [ 4443.169536] Lustre: Skipped 1 previous similar message [ 4446.557974] Lustre: server umount lustre-OST0000 complete [ 4451.296463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4451.302693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.332191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4453.348434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4453.358362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.465612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.483544] Lustre: Skipped 2 previous similar messages [ 4463.587328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4463.592322] Lustre: Skipped 1 previous similar message [ 4465.119155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4465.264457] Lustre: server umount lustre-MDT0000 complete [ 4469.085946] LustreError: 134818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445186 with bad export cookie 4733714408199792255 [ 4469.096782] LustreError: 134818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4469.097752] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4469.551282] Lustre: server umount lustre-MDT0001 complete [ 4480.043615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.636177] LustreError: 139014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4480.663606] LustreError: 139014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4480.782541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4486.117399] LustreError: 139015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.206052] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4491.236354] LustreError: 139014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4494.555843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.838455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4499.720283] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4508.367294] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4517.018141] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4526.065188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4526.465562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4533.862501] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4542.801257] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4547.981879] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4548.234627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4553.591704] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4553.840961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4563.932141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4564.136036] Lustre: lustre-OST0001: new disk, initializing [ 4564.140173] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4564.148905] Lustre: 142387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4564.297092] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4570.520953] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4573.789899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4573.804061] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4573.870498] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4578.684873] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4578.931600] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4584.083967] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4584.333823] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4591.012181] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9601c5d78800.ost_server_uuid 50 [ 4593.203874] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9601c5d78800.ost_server_uuid in IDLE state after 0 sec [ 4604.907960] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4604.908368] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4604.917835] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4604.919551] LustreError: Skipped 1 previous similar message [ 4604.938072] Lustre: Skipped 1 previous similar message [ 4605.195230] Lustre: server umount lustre-OST0001 complete [ 4610.017607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4610.023315] LustreError: 142874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.045405] Lustre: Skipped 1 previous similar message [ 4610.069353] LustreError: 142874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4615.140154] LustreError: 142874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.168139] LustreError: 142874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4615.878225] Lustre: server umount lustre-OST0000 complete [ 4620.257380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.377298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.383399] Lustre: Skipped 8 previous similar messages [ 4634.082362] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4634.328793] Lustre: server umount lustre-MDT0000 complete [ 4635.616492] LustreError: 139011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4635.648760] LustreError: 139011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4638.726645] LustreError: 138995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445356 with bad export cookie 4733714408199793137 [ 4638.736732] LustreError: 138995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4638.747153] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4639.391388] Lustre: server umount lustre-MDT0001 complete [ 4664.244051] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 4667.629729] Key type lgssc unregistered [ 4668.057465] LNet: 145395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4668.067138] LNetError: 145395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4668.104480] LNet: Removed LNI 192.168.204.103@tcp [ 4669.306170] Key type .llcrypt unregistered [ 4669.308096] Key type ._llcrypt unregistered [ 4717.093674] Key type ._llcrypt registered [ 4717.095947] Key type .llcrypt registered [ 4717.213932] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 4736.868353] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 4739.002744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4739.023745] alg: No test for adler32 (adler32-zlib) [ 4740.833676] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4741.312880] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 4743.031616] Key type lgssc registered [ 4744.279444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4786.790897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4786.825050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4788.112897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4788.142247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4788.242246] Lustre: lustre-MDT0000: new disk, initializing [ 4788.380028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.415945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4793.859775] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4806.181728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.399085] Lustre: 149555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4806.482514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4806.494360] Lustre: Skipped 1 previous similar message [ 4806.567377] Lustre: lustre-MDT0001: new disk, initializing [ 4806.707424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4806.775628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4806.796235] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4812.386233] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4823.490840] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4831.212861] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4839.590179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4839.848491] Lustre: lustre-OST0000: new disk, initializing [ 4839.855483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4839.869174] Lustre: 151522:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4840.015336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.608073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4845.625416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4845.741066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4846.461659] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4857.555936] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4862.404813] Lustre: Failing over lustre-OST0000 [ 4862.737368] Lustre: server umount lustre-OST0000 complete [ 4879.842622] Lustre: 147777:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445581/real 1788445581] req@000000007c3682b0 x1875320985651328/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788445597 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4879.873223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.905472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4882.283410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.327510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4882.346248] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4883.287290] Lustre: 147778:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445584/real 1788445584] req@00000000a2b44eb3 x1875320985652352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4883.314457] Lustre: 147778:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4884.333775] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4887.391103] Lustre: 147775:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445589/real 1788445589] req@00000000ae2dfd29 x1875320985653504/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4887.414336] Lustre: 147775:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4889.310626] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4890.963709] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4890.964578] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4893.023117] Lustre: 147776:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445594/real 1788445594] req@000000000b975ccf x1875320985654656/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4893.038428] Lustre: 147776:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4893.563342] Lustre: Failing over lustre-MDT0000 [ 4893.664202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4893.679338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.784778] Lustre: server umount lustre-MDT0000 complete [ 4899.813171] LustreError: 151779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4899.840603] LustreError: 151779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4901.144893] LustreError: 151530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.929711] LustreError: 149560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.960143] LustreError: 149560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4910.064829] LustreError: 150729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4910.097418] LustreError: 150729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4911.073158] Lustre: 147775:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445612/real 1788445612] req@0000000035e40875 x1875320985665152/t0(0) o400->MGC192.168.204.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788445628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4911.121455] Lustre: 147775:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4911.142985] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4915.610795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4916.193392] LustreError: 150729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.210764] LustreError: 150729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4920.309657] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3d8504ed3d7aca2a [ 4920.322302] Lustre: MGC192.168.204.103@tcp: Connection restored to 0@lo (at 0@lo) [ 4920.328610] Lustre: Skipped 1 previous similar message [ 4920.721446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4920.780948] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4921.385766] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4925.955448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4925.992878] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4926.088975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4926.516843] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 4936.673594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4936.677957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4936.686481] Lustre: Skipped 2 previous similar messages [ 4936.689886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4941.294614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4941.310796] Lustre: Skipped 2 previous similar messages [ 4942.859189] Lustre: server umount lustre-OST0000 complete [ 4951.522339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4951.541663] Lustre: Skipped 1 previous similar message [ 4951.550366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4951.555314] Lustre: Skipped 1 previous similar message [ 4953.825366] Lustre: server umount lustre-MDT0000 complete [ 4956.642972] LustreError: 150729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.683061] LustreError: 150729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4958.520451] LustreError: 149545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445676 with bad export cookie 4432954825201338922 [ 4958.528821] LustreError: 149545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4958.529898] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4959.146740] Lustre: server umount lustre-MDT0001 complete [ 4973.950112] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 4977.406435] Key type lgssc unregistered [ 4977.899211] LNet: 155187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4977.910752] LNetError: 155187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4977.928483] LNet: Removed LNI 192.168.204.103@tcp [ 4978.976881] Key type .llcrypt unregistered [ 4978.980562] Key type ._llcrypt unregistered [ 5022.445993] Key type ._llcrypt registered [ 5022.450569] Key type .llcrypt registered [ 5022.546723] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 5024.154189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5024.642287] alg: No test for adler32 (adler32-zlib) [ 5025.761734] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5026.059375] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 5027.735226] Key type lgssc registered [ 5029.126869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5038.792709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5038.816254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5040.510513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.844943] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5050.855881] LustreError: 156715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5050.883807] LustreError: 156715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5055.973170] LustreError: 156716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5057.277657] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5061.090292] LustreError: 156715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5061.941879] LustreError: 156744:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5061.956566] LustreError: 156744:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5061.963622] LustreError: 156744:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 5062.471123] Lustre: server umount lustre-MDT0000 complete [ 5080.180515] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 5092.031342] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 5139.544247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.827127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5139.861511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5139.938510] Lustre: lustre-MDT0000: new disk, initializing [ 5140.049237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5140.087306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5145.569514] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5157.438500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5157.558203] Lustre: 160049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5157.606384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5157.610137] Lustre: Skipped 1 previous similar message [ 5157.683828] Lustre: lustre-MDT0001: new disk, initializing [ 5157.777730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5157.831280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5157.840441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5162.451155] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5172.032947] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5178.168610] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5186.217816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5186.499727] Lustre: lustre-OST0000: new disk, initializing [ 5186.503841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5186.518948] Lustre: 162020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5186.651109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.815192] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5195.908813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5195.928562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5196.058549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5205.495467] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5211.732454] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5214.004034] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:32:08 (1788445928) [ 5500.896391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5500.905396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.924491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5503.459567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.471543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5503.485694] Lustre: Skipped 1 previous similar message [ 5503.496662] Lustre: Skipped 1 previous similar message [ 5504.849550] Lustre: server umount lustre-MDT0000 complete [ 5508.582536] LustreError: 160059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5508.606601] LustreError: 160059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5508.993598] LustreError: 160043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446226 with bad export cookie 28131836977537658 [ 5508.994353] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5509.000940] LustreError: 160043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5513.700969] LustreError: 161820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5513.703358] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5513.756994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5513.766232] Lustre: Skipped 1 previous similar message [ 5515.554142] Lustre: server umount lustre-MDT0001 complete [ 5534.176724] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5534.401814] Lustre: server umount lustre-OST0000 complete [ 5546.055297] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 5556.247442] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 5594.264610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5594.483971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5594.504676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5594.634591] Lustre: lustre-MDT0000: new disk, initializing [ 5594.754939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5594.772716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5599.429463] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5609.885465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5609.993585] Lustre: 166691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5610.080380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5610.084484] Lustre: Skipped 1 previous similar message [ 5610.179975] Lustre: lustre-MDT0001: new disk, initializing [ 5610.273491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5610.315133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5610.332166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5614.462449] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5624.534453] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5630.684613] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5638.412906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5638.696964] Lustre: lustre-OST0000: new disk, initializing [ 5638.705518] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5638.714034] Lustre: 168659:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5638.823647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5645.011833] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5647.934389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5647.939943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5648.064273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5654.563458] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5659.384878] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5659.604409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5664.752782] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5665.069267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5668.834957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5668.852594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5668.861340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5668.892659] Lustre: Skipped 1 previous similar message [ 5673.795317] Lustre: server umount lustre-OST0000 complete [ 5679.073638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5679.088048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5679.108623] Lustre: Skipped 2 previous similar messages [ 5684.768706] Lustre: server umount lustre-MDT0000 complete [ 5688.298076] LustreError: 169997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.340318] LustreError: 169997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5688.776968] LustreError: 166683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446406 with bad export cookie 28131836977696488 [ 5688.793297] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5688.797399] LustreError: 166683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5689.155189] Lustre: server umount lustre-MDT0001 complete [ 5697.412206] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:40:13 (1788446413) [ 5713.533503] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 5722.076465] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 5759.263206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.643195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5759.694436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5759.831756] Lustre: lustre-MDT0000: new disk, initializing [ 5759.941324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.965360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5765.002157] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5776.161836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5776.305920] Lustre: 173413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5776.703371] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5782.161554] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5791.930127] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5797.610345] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5804.822439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5805.136759] Lustre: lustre-OST0000: new disk, initializing [ 5805.143756] Lustre: Skipped 1 previous similar message [ 5805.154624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5805.159923] Lustre: Skipped 2 previous similar messages [ 5805.167486] Lustre: 175389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5812.067985] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5813.310155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5813.316792] Lustre: Skipped 1 previous similar message [ 5813.319990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5813.460689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5823.057405] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5834.215453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5834.216730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5834.220953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5834.220961] Lustre: Skipped 2 previous similar messages [ 5834.231212] Lustre: Skipped 2 previous similar messages [ 5834.264637] LustreError: Skipped 1 previous similar message [ 5837.727505] Lustre: server umount lustre-OST0000 complete [ 5843.427766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5848.031644] Lustre: server umount lustre-MDT0000 complete [ 5849.568889] LustreError: 173419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.582180] LustreError: 173419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5853.022378] LustreError: 173404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446570 with bad export cookie 28131836977697370 [ 5853.029296] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5853.033587] LustreError: 173404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5853.676513] Lustre: server umount lustre-MDT0001 complete [ 5863.053895] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:42:58 (1788446578) [ 5880.297292] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 5889.266947] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 5920.461883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.692726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5920.723927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5920.816186] Lustre: lustre-MDT0000: new disk, initializing [ 5920.914627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5920.920433] Lustre: Skipped 2 previous similar messages [ 5920.931834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5925.870590] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5936.485955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5936.603563] Lustre: 179714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5936.881256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5940.772802] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5950.079265] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5956.080152] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5963.820486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5964.290094] Lustre: 181693:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5971.561083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5971.656279] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 5971.685144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5981.912871] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5992.418625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5992.424714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.440046] LustreError: Skipped 1 previous similar message [ 5992.443881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5992.458732] Lustre: Skipped 3 previous similar messages [ 5992.483334] Lustre: Skipped 5 previous similar messages [ 5994.939401] Lustre: server umount lustre-OST0000 complete [ 6005.804540] Lustre: server umount lustre-MDT0000 complete [ 6007.780331] LustreError: 182040:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.808182] LustreError: 182040:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6009.443540] LustreError: 179709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446727 with bad export cookie 28131836977698455 [ 6009.457212] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6009.461334] LustreError: 179709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6009.866834] Lustre: server umount lustre-MDT0001 complete [ 6021.187965] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 6023.958834] Key type lgssc unregistered [ 6024.252625] LNet: 183552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6024.258649] LNetError: 183552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6024.284832] LNet: Removed LNI 192.168.204.103@tcp [ 6025.189460] Key type .llcrypt unregistered [ 6025.192279] Key type ._llcrypt unregistered [ 6048.366805] Key type ._llcrypt registered [ 6048.369386] Key type .llcrypt registered [ 6048.510454] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 6049.464278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6049.553621] alg: No test for adler32 (adler32-zlib) [ 6050.676900] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6050.902927] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 6052.567165] Key type lgssc registered [ 6053.865583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6062.043929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6062.072914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.454974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6068.362079] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6073.831645] LustreError: 184970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.853137] LustreError: 184970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6077.278913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.792528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6081.574195] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6090.164599] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6096.098142] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6103.034776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6103.603406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.726090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6110.421950] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6117.830321] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6143.460927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6143.461977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6143.475786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6143.492374] Lustre: Skipped 1 previous similar message [ 6146.050412] Lustre: server umount lustre-OST0000 complete [ 6149.600437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6149.606974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6149.615471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6149.622778] Lustre: Skipped 1 previous similar message [ 6153.697927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6153.700388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6155.838831] Lustre: server umount lustre-MDT0000 complete [ 6158.819354] LustreError: 184971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.835303] LustreError: 184971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6159.443634] LustreError: 184951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446877 with bad export cookie 11000972573741912084 [ 6159.447198] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6159.455836] LustreError: 184951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6159.844835] Lustre: server umount lustre-MDT0001 complete [ 6169.878670] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 6172.321912] Key type lgssc unregistered [ 6172.611389] LNet: 188532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6172.629258] LNetError: 188532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6172.653739] LNet: Removed LNI 192.168.204.103@tcp [ 6173.460276] Key type .llcrypt unregistered [ 6173.462088] Key type ._llcrypt unregistered [ 6194.332841] Key type ._llcrypt registered [ 6194.336134] Key type .llcrypt registered [ 6194.447390] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 6195.282302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6195.327271] alg: No test for adler32 (adler32-zlib) [ 6196.342865] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6196.499764] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 6198.143485] Key type lgssc registered [ 6199.194737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6205.824374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6205.843559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6207.206829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.987449] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6216.683445] LustreError: 189930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.713306] LustreError: 189930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6219.726698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6220.073990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6224.413356] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6231.633829] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6237.745357] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6244.888870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6245.149826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6251.325951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6252.516811] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6260.373994] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6271.971540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6271.982170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.999948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6273.326620] Lustre: lustre-OST0000: Not available for connect from 192.168.204.3@tcp (stopping) [ 6275.046164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6275.054594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6275.056895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6275.071314] Lustre: Skipped 1 previous similar message [ 6276.434975] Lustre: server umount lustre-OST0000 complete [ 6283.624311] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9601c912c000.ost_server_uuid 50 [ 6285.152925] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6299.619681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.623262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.641481] Lustre: Skipped 1 previous similar message [ 6304.736981] LustreError: 189931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.840936] Lustre: server umount lustre-MDT0000 complete [ 6308.225103] LustreError: 189910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447025 with bad export cookie 9239460284306918140 [ 6308.227528] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.234234] LustreError: 189910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6308.559430] Lustre: server umount lustre-MDT0001 complete [ 6316.236399] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:50:31 (1788447031) [ 6323.162777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6323.580192] LustreError: 193846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6323.595409] LustreError: 193846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6323.667289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6328.284458] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6328.806283] LustreError: 193847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6333.921213] LustreError: 193846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.273872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.642965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6340.761957] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6347.684223] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6353.202111] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6359.825680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6360.172182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.972934] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6368.376287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6372.717261] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6379.458516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6379.650579] Lustre: lustre-OST0001: new disk, initializing [ 6379.652506] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6379.657231] Lustre: 196687:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6379.725825] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6385.307490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6385.313705] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6385.490089] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6385.899599] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6397.365855] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6400.998867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6401.017586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6401.029604] Lustre: Skipped 2 previous similar messages [ 6411.247634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6411.255809] Lustre: Skipped 4 previous similar messages [ 6414.303151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6414.376088] Lustre: server umount lustre-OST0000 complete [ 6416.363217] LustreError: 195669:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6416.390093] LustreError: 195669:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6420.938791] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6421.248820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6426.125336] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6426.368843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6433.569050] LustreError: 195668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.591314] LustreError: 195668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6441.955401] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.965359] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6441.984986] Lustre: Skipped 1 previous similar message [ 6441.997017] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6441.997026] Lustre: Skipped 1 previous similar message [ 6446.357934] Lustre: server umount lustre-OST0001 complete [ 6452.192551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6452.225189] Lustre: Skipped 1 previous similar message [ 6457.050831] Lustre: server umount lustre-MDT0000 complete [ 6461.153715] LustreError: 193827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447178 with bad export cookie 9239460284306919211 [ 6461.160978] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6461.163168] LustreError: 193827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6461.599233] Lustre: server umount lustre-MDT0001 complete [ 6486.062303] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:53:21 (1788447201) [ 6493.377740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6493.741869] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6493.788989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6494.019443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.171175] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6508.492485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.609490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6508.660979] Lustre: 200131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6514.141671] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6521.643492] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6526.936547] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6533.458596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6533.618145] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6539.407619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6539.970412] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6547.510964] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6554.392718] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6554.483707] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6560.706436] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6568.123113] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6573.547159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6573.555878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6573.584169] Lustre: Skipped 1 previous similar message [ 6573.612449] Lustre: Skipped 4 previous similar messages [ 6585.311719] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6585.413979] Lustre: server umount lustre-OST0000 complete [ 6588.903857] LustreError: 201980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6588.918244] LustreError: 201980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6604.271090] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6604.283193] LustreError: Skipped 1 previous similar message [ 6606.236315] Lustre: server umount lustre-OST0001 complete [ 6611.427113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6611.438737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.457406] Lustre: Skipped 2 previous similar messages [ 6624.223176] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6624.368499] Lustre: server umount lustre-MDT0000 complete [ 6627.241994] LustreError: 200121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447345 with bad export cookie 9239460284306920618 [ 6627.243944] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6627.250478] LustreError: 200121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6627.523047] Lustre: server umount lustre-MDT0001 complete [ 6649.319364] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:56:04 (1788447364) [ 6664.700151] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 6672.995042] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 6706.467355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.724749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6706.765442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6706.854205] Lustre: lustre-MDT0000: new disk, initializing [ 6706.921570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6706.928112] Lustre: Skipped 3 previous similar messages [ 6706.955534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6712.019925] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6724.374890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6724.493982] Lustre: 208231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6724.548905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6724.552673] Lustre: Skipped 1 previous similar message [ 6724.629457] Lustre: lustre-MDT0001: new disk, initializing [ 6724.827680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6724.836626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6731.115185] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6741.959685] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6747.312461] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6755.323532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6755.832502] Lustre: lustre-OST0000: new disk, initializing [ 6755.837043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6755.849336] Lustre: 210202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6763.772124] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6765.642749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6765.657173] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6765.755384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6774.758155] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6778.955637] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6779.170855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6784.619136] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6784.985674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6791.650977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6791.658662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.672349] LustreError: Skipped 1 previous similar message [ 6791.676215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6791.688733] Lustre: Skipped 2 previous similar messages [ 6791.698405] Lustre: Skipped 14 previous similar messages [ 6793.478673] Lustre: server umount lustre-OST0000 complete [ 6804.341687] Lustre: server umount lustre-MDT0000 complete [ 6807.017702] LustreError: 208243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.038098] LustreError: 208243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6808.534168] LustreError: 208224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447526 with bad export cookie 9239460284306922193 [ 6808.551647] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6808.554089] LustreError: 208224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6808.927260] Lustre: server umount lustre-MDT0001 complete [ 6818.471368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6823.122436] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6831.113281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6834.488557] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6841.429736] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6848.249173] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6855.873575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6856.231369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6856.236833] Lustre: Skipped 4 previous similar messages [ 6861.651724] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6868.007497] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6872.219279] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6872.446353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6876.479837] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6876.713601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6878.689405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6878.700914] LustreError: Skipped 1 previous similar message [ 6884.719394] Lustre: server umount lustre-OST0000 complete [ 6891.332113] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6896.548652] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6926.903463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.421602] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6933.458370] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6933.467402] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6933.496387] Lustre: Skipped 1 previous similar message [ 6933.745823] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6941.461202] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6944.691268] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6953.965595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6953.975927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6953.977344] LustreError: Skipped 1 previous similar message [ 6953.979060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6953.979066] Lustre: Skipped 6 previous similar messages [ 6953.995983] Lustre: Skipped 5 previous similar messages [ 6955.662611] Lustre: server umount lustre-OST0000 complete [ 6966.328837] Lustre: server umount lustre-MDT0000 complete [ 6969.316038] LustreError: 212417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6969.339944] LustreError: 212417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6969.987693] LustreError: 213157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447687 with bad export cookie 9239460284306923075 [ 6969.993220] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6969.997482] LustreError: 213157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6970.336689] Lustre: server umount lustre-MDT0001 complete [ 6979.937166] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:01:35 (1788447695) [ 6986.658236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.928655] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 6998.691730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.474307] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7010.433772] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7016.209680] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7024.086573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7030.637589] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7037.854366] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7043.224682] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7043.453928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7047.486455] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7047.757166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7054.883421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7055.093735] Lustre: lustre-OST0001: new disk, initializing [ 7055.097483] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7055.104995] Lustre: 221771:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7061.890299] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7064.750254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7064.772126] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7064.855195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7070.580115] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7075.372971] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7075.681419] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7081.909892] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7082.125262] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7085.536429] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7085.562932] LustreError: Skipped 1 previous similar message [ 7090.745730] Lustre: server umount lustre-OST0001 complete [ 7097.237508] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7097.431259] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7101.814820] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7102.039275] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7130.454164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7130.710811] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7130.722267] Lustre: Skipped 5 previous similar messages [ 7135.728398] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7135.737783] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7135.738694] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7135.749914] Lustre: Skipped 1 previous similar message [ 7137.174590] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7143.379710] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7144.825430] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7161.311104] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7161.389234] Lustre: server umount lustre-OST0001 complete [ 7183.640807] Lustre: server umount lustre-OST0000 complete [ 7193.645467] Lustre: server umount lustre-MDT0000 complete [ 7197.591773] LustreError: 221770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447915 with bad export cookie 9239460284306924041 [ 7197.601882] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7197.604338] LustreError: 221770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7203.942356] Lustre: server umount lustre-MDT0001 complete [ 7231.168265] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:05:46 (1788447946) [ 7237.440227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7237.698690] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7237.719705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7242.997348] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7252.099662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.178660] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7252.202424] Lustre: 227017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7257.408865] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7267.542280] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7274.981776] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7283.045586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7283.304047] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7291.052089] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7298.692111] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7308.560806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7308.645729] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7315.476130] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7322.836975] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7324.781922] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7329.133858] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7329.320408] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7333.882606] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7334.164278] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7339.030310] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9601c3438000.ost_server_uuid 50 [ 7340.343954] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9601c3438000.ost_server_uuid in IDLE state after 0 sec [ 7341.230358] Lustre: Permanently deactivating lustre-OST0001 [ 7341.244228] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7344.602702] Lustre: Permanently reactivating lustre-OST0001 [ 7350.256078] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7350.257295] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7350.270027] LustreError: Skipped 6 previous similar messages [ 7350.271260] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7350.271267] Lustre: Skipped 21 previous similar messages [ 7350.296344] Lustre: Skipped 11 previous similar messages [ 7354.462741] Lustre: server umount lustre-OST0001 complete [ 7355.378801] LustreError: 228864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7355.394172] LustreError: 228864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 7365.398627] Lustre: server umount lustre-OST0000 complete [ 7375.504723] Lustre: server umount lustre-MDT0000 complete [ 7378.966531] LustreError: 227008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448096 with bad export cookie 9239460284306925329 [ 7378.969959] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7378.977231] LustreError: 227008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7379.264224] Lustre: server umount lustre-MDT0001 complete [ 7407.135497] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:08:42 (1788448122) [ 7415.385086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7415.823751] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7415.881673] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7415.901325] Lustre: Skipped 5 previous similar messages [ 7421.140172] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7431.181446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7431.406265] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7431.446794] Lustre: 233709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7436.695274] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7449.103743] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7461.106581] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7472.954306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7473.256110] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7473.297041] Lustre: Permanently deactivating lustre-OST0000 [ 7473.315633] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7480.405494] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7480.744365] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7488.635481] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7495.537871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.643569] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7502.696768] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7504.111176] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7510.938845] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7513.027653] Lustre: Permanently reactivating lustre-OST0000 [ 7513.045269] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7513.051619] Lustre: Skipped 2 previous similar messages [ 7518.822537] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7518.841032] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 38 seconds. I think it's dead, and I am evicting it. exp 000000002f5f3f54, cur 1788448236 deadline 1788448228 last 1788448198 [ 7518.922716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7546.335167] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7546.424389] Lustre: server umount lustre-OST0001 complete [ 7556.374990] Lustre: server umount lustre-OST0000 complete [ 7567.146709] Lustre: server umount lustre-MDT0000 complete [ 7571.980357] LustreError: 233703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448289 with bad export cookie 9239460284306927002 [ 7571.996996] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7572.808256] Lustre: server umount lustre-MDT0001 complete [ 7585.588168] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 7588.010324] Key type lgssc unregistered [ 7588.271305] LNet: 238598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7588.279362] LNetError: 238598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7588.303809] LNet: Removed LNI 192.168.204.103@tcp [ 7588.989166] Key type .llcrypt unregistered [ 7588.993076] Key type ._llcrypt unregistered [ 7614.543447] Key type ._llcrypt registered [ 7614.545743] Key type .llcrypt registered [ 7614.692759] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 7615.793470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7615.807374] alg: No test for adler32 (adler32-zlib) [ 7616.880795] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7617.134227] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 7618.863312] Key type lgssc registered [ 7620.540865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7632.135201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7632.186467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.800779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7638.397723] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7644.134325] LustreError: 240066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7644.146449] LustreError: 240066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7645.154349] LustreError: 240067:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.512221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7647.651384] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7647.714773] Lustre: 240056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7647.736528] Lustre: Permanently deactivating lustre-MDT0001 [ 7647.769477] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7648.312858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7652.879459] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7653.441707] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7660.712780] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7667.687563] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7674.705524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7675.062950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7680.190109] LustreError: 241905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7680.196492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7681.851788] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7685.616900] LustreError: 241905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7690.738109] LustreError: 241907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7691.042368] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7692.789732] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7699.318067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.681550] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7705.111030] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7705.116536] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7706.307984] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7714.206327] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7718.339990] Lustre: Permanently reactivating lustre-MDT0001 [ 7718.361511] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7718.370380] Lustre: Skipped 2 previous similar messages [ 7719.989578] Lustre: lustre-MDT0001: Client a5649ef2-03ed-4a67-8676-0d0f6089f275 (at 192.168.204.3@tcp) reconnecting [ 7726.196397] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.213863] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7726.222360] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7726.235294] LustreError: 243780:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7726.238526] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7726.251815] LustreError: 243780:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7726.251957] LustreError: 243780:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.103@tcp: cfg command failed: rc = -108 [ 7726.253638] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7726.253638] [ 7726.310105] LustreError: 240059:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7727.380579] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7729.300940] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7735.460502] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7735.743303] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7740.399973] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7740.642851] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7743.591060] Lustre: Permanently deactivating lustre-MDT0001 [ 7743.608876] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7743.619937] Lustre: Skipped 2 previous similar messages [ 7753.836387] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7760.865706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7760.871737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7760.880715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7762.402213] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7762.405211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7762.423703] Lustre: Skipped 1 previous similar message [ 7762.426197] Lustre: Skipped 1 previous similar message [ 7767.524850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7767.537377] Lustre: Skipped 5 previous similar messages [ 7772.651305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7772.664356] Lustre: Skipped 3 previous similar messages [ 7775.199092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7775.415699] Lustre: server umount lustre-MDT0000 complete [ 7777.772274] LustreError: 240810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7777.796717] LustreError: 240810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7779.693621] LustreError: 240048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448497 with bad export cookie 13483925507267830148 [ 7779.700947] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7779.711939] LustreError: 240048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7780.145775] Lustre: server umount lustre-MDT0001 complete [ 7786.163940] Lustre: server umount lustre-OST0000 complete [ 7801.105440] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:15:16 (1788448516) [ 7816.783347] Lustre: server umount lustre-OST0001 complete [ 7825.288305] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 7837.351079] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 7882.993094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7883.335542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7883.349892] Lustre: Skipped 2 previous similar messages [ 7883.407447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7883.569759] Lustre: lustre-MDT0000: new disk, initializing [ 7883.683989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.705651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7889.061464] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7901.895856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.090600] Lustre: 248349:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7902.180785] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7902.191786] Lustre: Skipped 1 previous similar message [ 7902.350104] Lustre: lustre-MDT0001: new disk, initializing [ 7902.459623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7902.487743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7902.498828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7908.077079] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7919.213855] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7925.473806] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7933.503302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7933.821583] Lustre: lustre-OST0000: new disk, initializing [ 7933.826363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7933.837321] Lustre: 250320:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7933.979094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7941.208280] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7943.759993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7943.770826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7943.896409] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7950.872839] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7953.913021] LustreError: 248357:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7963.002839] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7963.225750] Lustre: lustre-OST0001: new disk, initializing [ 7963.236449] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7963.251440] Lustre: 251434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7963.985431] LustreError: 248357:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7964.018814] LustreError: 250327:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7971.446266] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 7971.979894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7971.987368] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7972.086614] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7974.023814] LustreError: 250327:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7982.869478] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7984.814355] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7987.692297] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7987.692533] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7987.695249] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7987.707211] Lustre: Skipped 2 previous similar messages [ 7987.720032] LustreError: Skipped 1 previous similar message [ 7997.929544] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7997.939540] Lustre: Skipped 4 previous similar messages [ 8000.991175] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8001.089513] Lustre: server umount lustre-OST0001 complete [ 8003.055890] LustreError: 250305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8003.063410] LustreError: 250305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8018.446084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.453531] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8018.455706] Lustre: Skipped 1 previous similar message [ 8018.480849] Lustre: Skipped 1 previous similar message [ 8023.354443] Lustre: server umount lustre-OST0000 complete [ 8028.642886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.664442] Lustre: Skipped 1 previous similar message [ 8033.888250] Lustre: server umount lustre-MDT0000 complete [ 8037.835693] LustreError: 248342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448755 with bad export cookie 13483925507267832962 [ 8037.845690] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8037.848153] LustreError: 248342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8052.191142] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8052.530710] Lustre: server umount lustre-MDT0001 complete [ 8064.786937] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 8067.617755] Key type lgssc unregistered [ 8067.933361] LNet: 253476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8067.940405] LNetError: 253476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8068.966853] LNet: Removed LNI 192.168.204.103@tcp [ 8069.926837] Key type .llcrypt unregistered [ 8069.933660] Key type ._llcrypt unregistered [ 8110.954288] Key type ._llcrypt registered [ 8110.962968] Key type .llcrypt registered [ 8111.093948] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 8112.132982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8112.192112] alg: No test for adler32 (adler32-zlib) [ 8113.245672] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8113.462929] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 8115.119407] Key type lgssc registered [ 8116.194372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8123.793436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8123.818566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8125.057866] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8125.073638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8125.216946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.659471] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8138.860920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8138.947789] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8138.976056] Lustre: 255684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8139.081508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8139.197108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8143.643941] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8152.588447] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8159.243451] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8167.377657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8167.574198] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8167.986325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.135441] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8175.726848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8183.298792] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8242.144548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8242.158736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8242.170833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8242.665083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8242.672917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8247.777786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8247.795778] Lustre: Skipped 2 previous similar messages [ 8248.188287] Lustre: server umount lustre-OST0000 complete [ 8252.350949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8264.585914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8264.864623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8271.368657] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8273.072876] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8273.088122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8273.092962] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8279.572986] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8324.576298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8324.583237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8324.590369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8324.592090] LustreError: Skipped 1 previous similar message [ 8324.621997] Lustre: Skipped 1 previous similar message [ 8328.033634] Lustre: server umount lustre-OST0000 complete [ 8333.794375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8333.800914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8333.816095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8333.821640] Lustre: Skipped 1 previous similar message [ 8338.826273] Lustre: server umount lustre-MDT0000 complete [ 8343.143264] LustreError: 255677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449060 with bad export cookie 7789388923437464158 [ 8343.150219] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8343.155605] LustreError: 255677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8343.581654] Lustre: server umount lustre-MDT0001 complete [ 8355.543634] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 8358.398568] Key type lgssc unregistered [ 8358.733101] LNet: 261115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8358.745148] LNetError: 261115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8358.777516] LNet: Removed LNI 192.168.204.103@tcp [ 8359.818086] Key type .llcrypt unregistered [ 8359.824674] Key type ._llcrypt unregistered [ 8389.369624] Key type ._llcrypt registered [ 8389.373561] Key type .llcrypt registered [ 8389.507649] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 8391.206355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8391.240453] alg: No test for adler32 (adler32-zlib) [ 8392.449165] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8392.697301] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 8394.423214] Key type lgssc registered [ 8396.137745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8405.668939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8405.702412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.233732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.287612] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8416.750341] LustreError: 262538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.778041] LustreError: 262538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8421.866242] LustreError: 262537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8422.103286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.500330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.673365] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8436.408614] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8443.108053] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8450.763194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8451.256963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8452.274232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8458.935222] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8467.903707] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8498.657299] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8498.661299] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8498.678786] LustreError: Skipped 1 previous similar message [ 8498.682553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8498.729097] Lustre: Skipped 1 previous similar message [ 8501.168847] Lustre: server umount lustre-OST0000 complete [ 8508.898738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8508.915617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8508.925711] Lustre: Skipped 2 previous similar messages [ 8513.884953] Lustre: server umount lustre-MDT0000 complete [ 8514.018986] LustreError: 262532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8514.041712] LustreError: 262532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8517.947627] LustreError: 262519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449235 with bad export cookie 1325571278533734498 [ 8517.956278] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8517.956798] LustreError: 262519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8518.327668] Lustre: server umount lustre-MDT0001 complete [ 8529.975270] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 8532.920840] Key type lgssc unregistered [ 8533.261459] LNet: 266852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8533.271411] LNetError: 266852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8533.290312] LNet: Removed LNI 192.168.204.103@tcp [ 8534.103639] Key type .llcrypt unregistered [ 8534.106457] Key type ._llcrypt unregistered [ 8553.428249] Key type ._llcrypt registered [ 8553.430579] Key type .llcrypt registered [ 8553.623648] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 8554.600851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8554.767412] alg: No test for adler32 (adler32-zlib) [ 8556.015497] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8556.366239] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 8558.151334] Key type lgssc registered [ 8559.956774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8569.899400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8569.920394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8571.772222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8577.247634] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8582.117867] LustreError: 268075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8582.149882] LustreError: 268075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8587.242419] LustreError: 268076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8587.315612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.828515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8593.393187] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8602.645631] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8609.019425] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8616.821118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8617.272573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8624.932872] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8632.671452] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8641.846491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8656.592680] Lustre: server umount lustre-OST0000 complete [ 8662.501501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8662.524935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8662.527298] Lustre: Skipped 1 previous similar message [ 8662.546934] Lustre: Skipped 1 previous similar message [ 8667.616901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8667.637668] Lustre: Skipped 1 previous similar message [ 8671.840311] Lustre: server umount lustre-MDT0000 complete [ 8672.737896] LustreError: 268072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8672.768696] LustreError: 268072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8673.763414] Lustre: 267384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449375/real 1788449375] req@0000000004d21170 x1875324985965824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788449391 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8673.814148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8674.849071] Lustre: 267384:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449376/real 1788449376] req@000000001c5c3319 x1875324985966208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788449392 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8676.106367] LustreError: 268055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449393 with bad export cookie 17041102485039885 [ 8676.107468] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8676.135643] LustreError: 268055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8676.368944] Lustre: server umount lustre-MDT0001 complete [ 8687.384914] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 8690.799521] Key type lgssc unregistered [ 8691.151419] LNet: 271782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8691.169058] LNetError: 271782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8691.214635] LNet: Removed LNI 192.168.204.103@tcp [ 8692.121072] Key type .llcrypt unregistered [ 8692.125200] Key type ._llcrypt unregistered [ 8717.216181] Key type ._llcrypt registered [ 8717.217987] Key type .llcrypt registered [ 8717.304082] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 8718.360768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8718.442268] alg: No test for adler32 (adler32-zlib) [ 8719.606271] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8719.873071] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 8721.553828] Key type lgssc registered [ 8722.548730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8730.721556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8730.766264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8732.292351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8737.094941] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8742.882992] LustreError: 273202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8742.893899] LustreError: 273202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8745.184550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8745.706976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8750.305843] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8759.503373] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8765.358426] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8772.443660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8772.862802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8779.058139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8779.096739] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8788.710514] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8815.075846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8815.091035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8815.116408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8816.096904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8816.099023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8816.100596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8816.109386] Lustre: Skipped 1 previous similar message [ 8819.731912] Lustre: server umount lustre-OST0000 complete [ 8826.339424] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.343764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8826.354404] Lustre: Skipped 1 previous similar message [ 8826.364240] Lustre: Skipped 1 previous similar message [ 8830.048207] Lustre: server umount lustre-MDT0000 complete [ 8831.456674] LustreError: 273197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.475418] LustreError: 273197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8834.106093] LustreError: 273182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449551 with bad export cookie 3393396919616264344 [ 8834.106830] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8834.113698] LustreError: 273182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8834.479758] Lustre: server umount lustre-MDT0001 complete [ 8845.427727] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 8849.628213] Key type lgssc unregistered [ 8850.037838] LNet: 277510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8850.050113] LNetError: 277510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8850.063711] LNet: Removed LNI 192.168.204.103@tcp [ 8851.047202] Key type .llcrypt unregistered [ 8851.049637] Key type ._llcrypt unregistered [ 8868.764312] Key type ._llcrypt registered [ 8868.770788] Key type .llcrypt registered [ 8868.889589] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 8870.550065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8870.604675] alg: No test for adler32 (adler32-zlib) [ 8871.755821] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8872.087385] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 8873.847256] Key type lgssc registered [ 8874.923619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8882.699203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8882.718963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.144480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8888.597120] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8893.413302] LustreError: 278748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8893.462066] LustreError: 278748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8898.383289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8898.533652] LustreError: 278750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8898.845287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8903.586039] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8911.866490] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8918.052364] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8925.039789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8925.354756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8928.443249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8931.642460] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 8940.852912] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8964.577471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8964.578868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8964.596903] LustreError: Skipped 1 previous similar message [ 8964.601157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8964.667486] Lustre: Skipped 1 previous similar message [ 8966.902698] Lustre: server umount lustre-OST0000 complete [ 8974.817699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8974.821058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8974.825309] Lustre: Skipped 1 previous similar message [ 8974.830755] Lustre: Skipped 1 previous similar message [ 8978.548653] Lustre: server umount lustre-MDT0000 complete [ 8979.938743] LustreError: 278761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8979.961811] LustreError: 278761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8982.153746] LustreError: 278715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449699 with bad export cookie 7522525891497322507 [ 8982.190401] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8982.439767] Lustre: server umount lustre-MDT0001 complete [ 8993.889887] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 8996.715812] Key type lgssc unregistered [ 8996.982210] LNet: 282452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8996.991211] LNetError: 282452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8998.065449] LNet: Removed LNI 192.168.204.103@tcp [ 8998.919331] Key type .llcrypt unregistered [ 8998.922308] Key type ._llcrypt unregistered [ 9009.976828] Key type ._llcrypt registered [ 9009.982529] Key type .llcrypt registered [ 9010.122369] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9027.316356] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 9040.015261] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 9041.029881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9041.134154] alg: No test for adler32 (adler32-zlib) [ 9042.189965] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9042.421564] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 9044.071192] Key type lgssc registered [ 9045.143555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9083.804352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9083.837487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.299733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9085.334672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9085.410713] Lustre: lustre-MDT0000: new disk, initializing [ 9085.479660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9085.494568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9091.458406] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9102.826105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9102.981532] Lustre: 285985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9103.043960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9103.050242] Lustre: Skipped 1 previous similar message [ 9103.193600] Lustre: lustre-MDT0001: new disk, initializing [ 9103.322407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9103.362080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9103.370779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9107.510838] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9117.077572] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9122.833572] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9129.235268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9129.478618] Lustre: lustre-OST0000: new disk, initializing [ 9129.486732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9129.501234] Lustre: 287954:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9129.602954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9135.061650] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9136.744932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9136.755209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9136.903254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9144.333233] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9148.919569] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9149.122461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9153.731853] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9153.934144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9157.603268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9157.611514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9157.623356] Lustre: Skipped 1 previous similar message [ 9157.631603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9157.644420] Lustre: Skipped 1 previous similar message [ 9162.003971] Lustre: server umount lustre-OST0000 complete [ 9167.845300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9167.872298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9167.876935] Lustre: Skipped 1 previous similar message [ 9172.417039] Lustre: server umount lustre-MDT0000 complete [ 9176.040533] LustreError: 287964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449893 with bad export cookie 11861087169049452487 [ 9176.045449] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9176.054299] LustreError: 287964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9176.458107] Lustre: server umount lustre-MDT0001 complete [ 9184.888723] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:38:20 (1788449900) [ 9191.569201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.031662] LustreError: 290366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9192.048286] LustreError: 290366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9192.102163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9196.333463] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9197.542099] LustreError: 290367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9201.635643] LustreError: 290366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9204.640572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9204.965884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9209.394750] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9217.433535] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9222.871934] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9229.597482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9229.740614] LustreError: 292187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9229.965931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9236.315897] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9244.626532] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9255.904829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9255.919621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9255.944851] Lustre: Skipped 1 previous similar message [ 9255.965244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9256.945970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9259.365412] Lustre: server umount lustre-OST0000 complete [ 9266.656428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9266.660179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9266.675726] Lustre: Skipped 1 previous similar message [ 9266.679832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9266.685218] Lustre: Skipped 2 previous similar messages [ 9272.293405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9272.301107] Lustre: Skipped 3 previous similar messages [ 9277.407182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9277.607894] Lustre: server umount lustre-MDT0000 complete [ 9281.242628] LustreError: 290346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449999 with bad export cookie 11861087169049453362 [ 9281.259308] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9281.793678] Lustre: server umount lustre-MDT0001 complete [ 9293.175564] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 9296.244396] Key type lgssc unregistered [ 9296.714679] LNet: 293905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9296.731639] LNetError: 293905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9296.776198] LNet: Removed LNI 192.168.204.103@tcp [ 9297.854510] Key type .llcrypt unregistered [ 9297.857375] Key type ._llcrypt unregistered [ 9327.440234] Key type ._llcrypt registered [ 9327.442995] Key type .llcrypt registered [ 9327.558716] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 9343.213628] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 9344.489490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9344.511223] alg: No test for adler32 (adler32-zlib) [ 9345.535289] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9345.780677] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 9347.447617] Key type lgssc registered [ 9349.093799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9396.623197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9396.677791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9397.957574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9397.978697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9398.074754] Lustre: lustre-MDT0000: new disk, initializing [ 9398.148728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9398.161893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9403.100477] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9415.442659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9415.740253] Lustre: 297735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9415.807738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9415.813669] Lustre: Skipped 1 previous similar message [ 9415.957403] Lustre: lustre-MDT0001: new disk, initializing [ 9416.136785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9416.214915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9416.232101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9421.216554] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9432.909376] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9439.255830] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9447.352617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9447.686511] Lustre: lustre-OST2710: new disk, initializing [ 9447.694402] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9447.704725] Lustre: 299703:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9447.842541] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9454.182729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9454.198710] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9454.383605] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9455.542807] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9466.702742] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9475.607924] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9475.788766] Lustre: lustre-OST03e8: new disk, initializing [ 9475.796388] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9475.807064] Lustre: 300767:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9475.936260] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9482.393209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9482.410168] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9482.574116] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9483.169894] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9492.417800] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9501.046892] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9501.284848] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9507.267614] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9507.501623] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9513.374064] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9513.630435] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9519.470432] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9519.766992] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9525.235967] Lustre: server umount lustre-MDT0000 complete [ 9528.805952] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9528.807583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9528.814518] Lustre: Skipped 3 previous similar messages [ 9529.440995] LustreError: 297728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450247 with bad export cookie 10502774590939712665 [ 9529.443483] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9529.454068] LustreError: 297728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9529.857916] Lustre: server umount lustre-MDT0001 complete [ 9543.621448] Lustre: server umount lustre-OST2710 complete [ 9557.927169] Lustre: server umount lustre-OST03e8 complete [ 9580.729396] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [ 9592.555841] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 9630.222288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9630.500317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9630.557805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9630.668155] Lustre: lustre-MDT0000: new disk, initializing [ 9630.765403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9630.796721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9634.643661] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9646.318659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9646.568662] Lustre: 306020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9646.626902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9646.630656] Lustre: Skipped 1 previous similar message [ 9646.710497] Lustre: lustre-MDT0001: new disk, initializing [ 9646.842357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9646.890249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9646.907723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9653.954647] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9665.051235] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9671.168304] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9679.222061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9679.609084] Lustre: lustre-OST0000: new disk, initializing [ 9679.623847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9679.636192] Lustre: 307988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9679.777426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9687.163517] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9689.175207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9689.185550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9689.374390] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9696.808395] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9701.747730] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9701.958297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9706.578237] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9706.789339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9710.048411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9710.049307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.053808] LustreError: Skipped 1 previous similar message [ 9710.069262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9714.820550] Lustre: server umount lustre-OST0000 complete [ 9718.753440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9718.762733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9718.778741] Lustre: Skipped 1 previous similar message [ 9718.786264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9718.789996] Lustre: Skipped 1 previous similar message [ 9720.291442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9720.296750] Lustre: Skipped 1 previous similar message [ 9724.498667] Lustre: server umount lustre-MDT0000 complete [ 9725.409809] LustreError: 307183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9725.424559] LustreError: 307183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9728.286448] LustreError: 306012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450446 with bad export cookie 10502774590939714345 [ 9728.294409] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9728.302427] LustreError: 306012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9728.802510] Lustre: server umount lustre-MDT0001 complete [ 9738.052898] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:47:33 (1788450453) [ 9739.841767] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9741.710943] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:47:37 (1788450457) [ 9745.496206] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 9775.567259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.850324] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9775.880235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9776.053933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9783.507977] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9793.639586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9793.776646] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9793.822721] Lustre: 311845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9798.809842] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9821.166511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9821.448217] Lustre: Denying initial registration attempt from nid 192.168.204.103@tcp, specified as failover [ 9821.459647] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9821.477716] LustreError: 313299:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9821.500905] LustreError: 313299:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9821.515122] LustreError: 313299:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9821.641390] Lustre: server umount lustre-OST0000 complete [ 9821.651270] LustreError: 313299:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9829.347328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9829.348186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9829.361219] Lustre: Skipped 2 previous similar messages [ 9829.374146] Lustre: Skipped 1 previous similar message [ 9834.467264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9834.480553] Lustre: Skipped 1 previous similar message [ 9835.112370] Lustre: server umount lustre-MDT0000 complete [ 9839.074514] LustreError: 311837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450556 with bad export cookie 10502774590939715220 [ 9839.075732] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9839.088452] LustreError: 311837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9839.646614] Lustre: server umount lustre-MDT0001 complete [ 9851.276768] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [ 9854.640932] Key type lgssc unregistered [ 9854.969072] LNet: 314425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9854.983730] LNetError: 314425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9855.017514] LNet: Removed LNI 192.168.204.103@tcp [ 9856.101578] Key type .llcrypt unregistered [ 9856.105122] Key type ._llcrypt unregistered [ 9869.827918] Key type ._llcrypt registered [ 9869.831159] Key type .llcrypt registered [ 9869.961876] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 9871.142386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9871.275147] alg: No test for adler32 (adler32-zlib) [ 9872.451499] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9872.810564] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [ 9874.599294] Key type lgssc registered [ 9876.002411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9924.742294] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [ 9934.665308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9934.690691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9936.070736] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9936.106224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9936.289553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9941.414893] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9951.068583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9951.255866] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9951.311237] Lustre: 317188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9951.753263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9957.147121] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [ 9965.898512] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9973.098963] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9994.877949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9995.262052] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9995.622796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10002.519758] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10011.305755] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10017.761258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10017.766941] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10017.793898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10017.810784] Lustre: Skipped 1 previous similar message [10017.831932] Lustre: Skipped 1 previous similar message [10019.848774] Lustre: server umount lustre-OST0000 complete [10028.002222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10028.007670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10028.038427] Lustre: Skipped 1 previous similar message [10028.052243] Lustre: Skipped 1 previous similar message [10030.988293] Lustre: server umount lustre-MDT0000 complete [10033.121522] LustreError: 317199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10033.142855] LustreError: 317199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10034.879774] LustreError: 317180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450752 with bad export cookie 5286624426302183231 [10034.884335] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10034.893989] LustreError: 317180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10035.417169] Lustre: server umount lustre-MDT0001 complete [10047.719241] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [10050.668581] Key type lgssc unregistered [10051.031452] LNet: 320768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10051.055573] LNetError: 320768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10051.078075] LNet: Removed LNI 192.168.204.103@tcp [10051.950652] Key type .llcrypt unregistered [10051.956655] Key type ._llcrypt unregistered [10078.762743] Key type ._llcrypt registered [10078.765111] Key type .llcrypt registered [10078.872702] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [10079.897347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10080.022137] alg: No test for adler32 (adler32-zlib) [10081.162793] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10081.429456] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [10083.135226] Key type lgssc registered [10084.241660] Lustre: Echo OBD driver; http://www.lustre.org/ [10091.819690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10091.851387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.280781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10097.550906] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10103.782084] LustreError: 322200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10103.801752] LustreError: 322200:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10104.312797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10104.602882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10108.104071] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10114.784453] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10120.654790] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10130.478405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10131.011878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10138.511794] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10147.389584] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10162.532380] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.3@tcp (stopping) [10164.709614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10164.715661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10164.729142] Lustre: Skipped 2 previous similar messages [10164.736256] Lustre: Skipped 1 previous similar message [10167.585722] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.3@tcp (stopping) [10168.619734] Lustre: server umount lustre-MDT0000 complete [10169.824367] LustreError: 322926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10169.845892] LustreError: 322926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10172.541712] LustreError: 322181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450890 with bad export cookie 4499231152461517831 [10172.545984] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10172.556948] LustreError: 322181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10172.703323] LustreError: 322195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10172.738964] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.3@tcp (stopping) [10174.946468] LustreError: 322194:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10174.948640] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10177.826148] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.3@tcp (stopping) [10177.836519] Lustre: Skipped 1 previous similar message [10178.987684] Lustre: server umount lustre-MDT0001 complete [10186.271553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10195.786077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10197.483653] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3e707ae52cd730f5 [10197.504128] Lustre: MGC192.168.204.103@tcp: Connection restored to 0@lo (at 0@lo) [10197.854577] LustreError: 324029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10197.878839] LustreError: 324029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10197.935705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10198.034432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10203.250874] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10207.390032] LustreError: 325818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10207.399225] LustreError: 325818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10208.172736] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10211.944124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10212.288880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10216.420609] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10217.457976] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10224.084238] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10225.695244] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10231.483917] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10232.848791] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10237.922359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10237.950491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10237.959203] Lustre: Skipped 2 previous similar messages [10252.255548] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10252.435510] Lustre: server umount lustre-OST0000 complete [10258.400865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10258.407675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10258.421674] Lustre: Skipped 2 previous similar messages [10258.421996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10258.455462] Lustre: Skipped 5 previous similar messages [10270.175151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10270.406464] Lustre: server umount lustre-MDT0000 complete [10273.768504] LustreError: 325817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.790291] LustreError: 325817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10274.735926] LustreError: 322181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450992 with bad export cookie 4499231152461525237 [10274.739936] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10274.757403] LustreError: 322181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10275.127971] Lustre: server umount lustre-MDT0001 complete [10288.979445] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [10292.329633] Key type lgssc unregistered [10292.667622] LNet: 328496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10292.674400] LNetError: 328496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10292.690924] LNet: Removed LNI 192.168.204.103@tcp [10293.693964] Key type .llcrypt unregistered [10293.695838] Key type ._llcrypt unregistered [10325.204281] Key type ._llcrypt registered [10325.206956] Key type .llcrypt registered [10325.340915] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [10326.295332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10326.469836] alg: No test for adler32 (adler32-zlib) [10327.530182] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10327.756127] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [10329.455820] Key type lgssc registered [10330.743651] Lustre: Echo OBD driver; http://www.lustre.org/ [10339.321607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10339.342034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10340.839396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10345.413746] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10351.078192] LustreError: 329911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10351.100866] LustreError: 329911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10354.917377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10355.275668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10360.201434] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10369.300940] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10376.752557] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10380.770123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10380.801816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10380.845822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10384.866203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10384.874581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10386.557339] Lustre: server umount lustre-MDT0000 complete [10389.987614] LustreError: 329911:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10390.002315] LustreError: 329911:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10390.720302] LustreError: 329893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451108 with bad export cookie 9860690646156232675 [10390.725482] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10390.737293] LustreError: 329893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10391.317783] Lustre: server umount lustre-MDT0001 complete [10406.297210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.798774] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10406.821498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10407.024017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10412.076799] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10420.846287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10420.999183] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10421.049447] Lustre: 332501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10421.334646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10426.197681] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10433.931816] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10440.687404] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10450.546491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10450.773735] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10450.785421] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10450.798486] LustreError: 334388:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10450.806707] LustreError: 334388:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10450.812307] LustreError: 334388:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10450.907610] Lustre: server umount lustre-OST0000 complete [10450.910677] LustreError: 334388:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [10461.550141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10461.815768] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10462.008187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10468.641647] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10469.451900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10476.869771] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10487.582311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10487.674029] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10487.749503] Lustre: lustre-OST0001: new disk, initializing [10487.752682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10487.765369] Lustre: 335931:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10487.876315] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10494.159603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10494.166065] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10494.332050] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10496.345414] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10505.622792] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10509.827387] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10509.832259] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10509.843387] LustreError: Skipped 1 previous similar message [10509.862045] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10509.887328] Lustre: Skipped 2 previous similar messages [10514.917572] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10514.938772] Lustre: Skipped 1 previous similar message [10515.469844] Lustre: server umount lustre-OST0001 complete [10520.042578] LustreError: 336200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10520.049016] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10520.072140] Lustre: Skipped 1 previous similar message [10520.077236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10520.084129] Lustre: Skipped 1 previous similar message [10525.157862] LustreError: 334855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10525.172879] LustreError: 334855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10530.286543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10530.289511] LustreError: 334857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.306393] Lustre: Skipped 3 previous similar messages [10530.349886] LustreError: 334857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10534.367604] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10534.538937] Lustre: server umount lustre-OST0000 complete [10538.976795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10538.981191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10538.989918] Lustre: Skipped 1 previous similar message [10544.840889] Lustre: server umount lustre-MDT0000 complete [10545.634446] LustreError: 332509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10545.659973] LustreError: 332509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10549.182683] LustreError: 332494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451266 with bad export cookie 9860690646156233228 [10549.192421] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10549.194308] LustreError: 332494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10549.526261] Lustre: server umount lustre-MDT0001 complete [10561.596447] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [10564.700483] Key type lgssc unregistered [10565.028214] LNet: 337907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10565.035842] LNetError: 337907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10565.053234] LNet: Removed LNI 192.168.204.103@tcp [10566.008950] Key type .llcrypt unregistered [10566.011147] Key type ._llcrypt unregistered [10645.425615] Key type ._llcrypt registered [10645.429399] Key type .llcrypt registered [10645.536981] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [10662.105366] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [10663.779142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10663.872486] alg: No test for adler32 (adler32-zlib) [10665.044554] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10665.430562] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [10667.215269] Key type lgssc registered [10668.921185] Lustre: Echo OBD driver; http://www.lustre.org/ [10709.348412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10709.368441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10710.675344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10710.696978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10710.759894] Lustre: lustre-MDT0000: new disk, initializing [10710.804568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10710.821620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10716.249885] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10728.250629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10728.484529] Lustre: 342842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10728.622234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10728.630984] Lustre: Skipped 1 previous similar message [10728.771366] Lustre: lustre-MDT0001: new disk, initializing [10729.043903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10729.116056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10729.134434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10734.076377] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10745.171720] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10751.601668] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10759.199713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10759.559781] Lustre: lustre-OST0000: new disk, initializing [10759.563524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10759.593648] Lustre: 344809:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10759.704196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10766.720395] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10768.483196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10768.495323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10768.642655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10777.146803] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10782.139760] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10782.320039] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10787.942945] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10788.288664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10794.466925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10794.470439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10794.480798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10794.501608] Lustre: Skipped 1 previous similar message [10794.521028] Lustre: Skipped 1 previous similar message [10796.779992] Lustre: server umount lustre-OST0000 complete [10804.718504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10804.722264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10804.741107] Lustre: Skipped 1 previous similar message [10807.560280] Lustre: server umount lustre-MDT0000 complete [10812.010375] LustreError: 342833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451529 with bad export cookie 11901408447068318127 [10812.017868] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10812.041371] LustreError: 342833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10812.605747] Lustre: server umount lustre-MDT0001 complete [10823.560572] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:05:38 (1788451538) [10832.416780] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:05:47 (1788451547) [10840.641742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10841.067938] LustreError: 347568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10841.085634] LustreError: 347568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10841.142892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10844.443675] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10846.180216] LustreError: 347569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.275223] LustreError: 347568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10853.683847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10854.040042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10860.827752] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10871.494532] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10878.194907] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10885.457445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10885.876144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10892.396788] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10900.950521] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10909.192499] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10911.739187] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10913.887354] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10916.189982] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10918.617943] Lustre: Failing over lustre-MDT0000 [10919.174428] Lustre: server umount lustre-MDT0000 complete [10919.907049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10919.913303] Lustre: Skipped 1 previous similar message [10919.914212] LustreError: 347564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10919.942600] LustreError: 347564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10925.029399] LustreError: 347569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10925.050106] LustreError: 347569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10933.540142] LustreError: 347563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10933.561735] LustreError: 347563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10935.269758] Lustre: 341066:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451637/real 1788451637] req@000000000a6ea32d x1875327197455488/t0(0) o400->MGC192.168.204.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788451653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10935.307992] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10940.263200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10946.527162] LustreError: 350770:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10946.533381] LustreError: 341063:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009b81b6e6 x1875327197460480/t0(0) o250->MGC192.168.204.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10946.969444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10947.056764] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10949.928032] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10952.190036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10952.232488] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10952.306830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10953.715095] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [10965.130263] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10966.948616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10971.078501] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10973.543898] Lustre: Failing over lustre-MDT0000 [10973.804481] Lustre: server umount lustre-MDT0000 complete [10975.585366] LustreError: 347563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10975.633778] LustreError: 347563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [10977.760665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10977.763636] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10977.812883] Lustre: Skipped 2 previous similar messages [10993.615171] Lustre: 341067:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451695/real 1788451695] req@00000000839d760c x1875327197478528/t0(0) o400->MGC192.168.204.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788451711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10993.656039] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10994.986920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11003.875024] LustreError: 341063:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c0dcd139 x1875327197483264/t0(0) o250->MGC192.168.204.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11003.919847] LustreError: 341063:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 1 previous similar message [11004.367707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11004.437487] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11006.239345] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11009.081928] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11009.521088] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11009.526142] Lustre: Skipped 2 previous similar messages [11009.555232] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11009.598801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11020.866714] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11023.020387] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11035.104379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11035.119885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11035.126577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11035.137746] Lustre: Skipped 2 previous similar messages [11035.158152] Lustre: Skipped 3 previous similar messages [11037.304047] Lustre: server umount lustre-MDT0000 complete [11040.234083] LustreError: 347568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11040.260187] LustreError: 347568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [11042.965426] LustreError: 347550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451760 with bad export cookie 11901408447068320843 [11042.983087] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11042.995562] LustreError: 347550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11043.942789] Lustre: server umount lustre-MDT0001 complete [11059.548378] Lustre: server umount lustre-OST0000 complete [11078.809195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11079.438832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11084.987849] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11095.081206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11099.891278] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11108.859787] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11115.624321] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11123.362272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11124.926802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11131.576990] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11140.305713] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11145.697663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11145.700770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11145.705109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11147.465157] Lustre: server umount lustre-OST0000 complete [11151.842922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11151.859496] LustreError: Skipped 1 previous similar message [11151.865863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11151.870360] Lustre: Skipped 1 previous similar message [11161.060318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11161.071780] Lustre: Skipped 3 previous similar messages [11166.177613] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11166.414140] Lustre: server umount lustre-MDT0000 complete [11170.493243] LustreError: 353899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451888 with bad export cookie 11901408447068321585 [11170.497673] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11170.512653] LustreError: 353899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11171.037453] Lustre: server umount lustre-MDT0001 complete [11183.996571] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [11186.997271] Key type lgssc unregistered [11187.299520] LNet: 357454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11187.311864] LNetError: 357454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11187.353274] LNet: Removed LNI 192.168.204.103@tcp [11188.321839] Key type .llcrypt unregistered [11188.324793] Key type ._llcrypt unregistered [11216.512714] Key type ._llcrypt registered [11216.514365] Key type .llcrypt registered [11216.608656] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [11217.291986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11217.426428] alg: No test for adler32 (adler32-zlib) [11218.465995] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11218.681652] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [11220.559680] Key type lgssc registered [11222.480576] Lustre: Echo OBD driver; http://www.lustre.org/ [11234.028480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11234.058548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11235.472431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11240.895722] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11246.050882] LustreError: 358975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11246.076338] LustreError: 358975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11251.057155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11251.170808] LustreError: 358976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11251.609368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11257.060562] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11266.083923] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11272.607269] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11284.155232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11284.795713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11293.764837] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11302.750675] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11305.262858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11318.241615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11318.249387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11318.263353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11320.431046] Lustre: server umount lustre-MDT0000 complete [11320.808667] LustreError: 358975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11320.811139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11320.840829] Lustre: Skipped 1 previous similar message [11324.028382] LustreError: 358957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452041 with bad export cookie 12834410571827083078 [11324.043910] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11324.308868] Lustre: server umount lustre-MDT0001 complete [11339.320193] Lustre: server umount lustre-OST0000 complete [11354.805134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11355.320085] LustreError: 362709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11355.341255] LustreError: 362709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11355.430610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11359.385522] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11360.741157] LustreError: 362710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11370.851852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11370.977415] LustreError: 362709:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11370.993127] LustreError: 362709:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11371.320391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11377.195050] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11385.952626] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11393.009931] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11400.213654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11400.600582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11401.656552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11407.491987] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11415.419412] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11427.811906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11427.817388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11427.829759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11431.399532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11431.403578] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11433.249853] Lustre: server umount lustre-MDT0000 complete [11436.520174] LustreError: 363652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.555311] LustreError: 363652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11438.998789] LustreError: 362690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452156 with bad export cookie 12834410571827084184 [11439.002449] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11439.015721] LustreError: 362690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11439.587448] Lustre: server umount lustre-MDT0001 complete [11455.301485] Lustre: server umount lustre-OST0000 complete [11475.991437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11476.996068] LustreError: 366623:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11477.186989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11482.812410] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11492.686096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11498.490026] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11507.939142] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11513.867448] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11522.111576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11522.437234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11522.445199] Lustre: Skipped 1 previous similar message [11529.069911] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11537.521817] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11547.067865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11557.345376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11557.361105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11557.367320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11557.388422] Lustre: Skipped 2 previous similar messages [11557.408458] Lustre: Skipped 3 previous similar messages [11560.634372] Lustre: server umount lustre-OST0000 complete [11565.026395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11565.045579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11567.593845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11567.618584] Lustre: Skipped 1 previous similar message [11577.829462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11577.834758] Lustre: Skipped 5 previous similar messages [11579.359437] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11579.502207] Lustre: server umount lustre-MDT0000 complete [11582.968718] LustreError: 367352:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11583.005925] LustreError: 367352:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11583.861889] LustreError: 369428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452301 with bad export cookie 12834410571827085213 [11583.875457] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11583.895247] LustreError: 369428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11584.305188] Lustre: server umount lustre-MDT0001 complete [11598.432680] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [11602.133383] Key type lgssc unregistered [11602.625499] LNet: 370185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11602.643417] LNetError: 370185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11603.690965] LNet: Removed LNI 192.168.204.103@tcp [11605.018724] Key type .llcrypt unregistered [11605.021585] Key type ._llcrypt unregistered [11636.491761] Key type ._llcrypt registered [11636.498364] Key type .llcrypt registered [11636.642702] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [11638.061463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11638.091146] alg: No test for adler32 (adler32-zlib) [11639.213790] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11639.574892] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [11641.320699] Key type lgssc registered [11642.720772] Lustre: Echo OBD driver; http://www.lustre.org/ [11652.808073] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11652.827822] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11652.835343] LustreError: 371629:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11652.851353] LustreError: 371629:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11652.865833] LustreError: 371629:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11652.876553] LustreError: 371629:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11652.892234] LustreError: 371629:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11660.214863] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11660.220200] LustreError: 372026:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11660.234894] LustreError: 372026:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11660.244167] LustreError: 372026:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11660.249849] LustreError: 372026:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11660.255540] LustreError: 372026:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11681.298815] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [11684.104489] Key type lgssc unregistered [11684.461912] LNet: 373033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11684.468355] LNetError: 373033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11684.499872] LNet: Removed LNI 192.168.204.103@tcp [11685.466460] Key type .llcrypt unregistered [11685.470808] Key type ._llcrypt unregistered [11705.670358] Key type ._llcrypt registered [11705.672554] Key type .llcrypt registered [11705.777278] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [11721.585228] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [11722.693581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11722.849859] alg: No test for adler32 (adler32-zlib) [11723.945225] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11724.258282] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [11725.984851] Key type lgssc registered [11727.042194] Lustre: Echo OBD driver; http://www.lustre.org/ [11767.245896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11767.271368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.696926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11768.742317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11768.927477] Lustre: lustre-MDT0000: new disk, initializing [11769.040205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11769.058524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11774.536908] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11787.047681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11787.170846] Lustre: 376193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11787.212763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11787.216162] Lustre: Skipped 1 previous similar message [11787.321079] Lustre: lustre-MDT0001: new disk, initializing [11787.443052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11787.483179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11787.496159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11792.843800] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11803.215490] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11809.972762] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11816.898096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11817.169771] Lustre: lustre-OST0000: new disk, initializing [11817.176376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11817.188331] Lustre: 378158:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11817.296393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11822.669474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11822.684142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11822.812498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11823.660981] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11833.532977] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11838.676118] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11838.893779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11844.738666] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11845.058730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11848.673319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.695430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11848.696069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11848.710638] Lustre: Skipped 1 previous similar message [11853.929942] Lustre: server umount lustre-OST0000 complete [11858.913304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11858.915108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11858.936281] Lustre: Skipped 1 previous similar message [11858.941547] Lustre: Skipped 1 previous similar message [11864.032115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11864.043289] Lustre: Skipped 1 previous similar message [11869.153251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11869.171313] Lustre: Skipped 1 previous similar message [11873.249174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11873.728226] Lustre: server umount lustre-MDT0000 complete [11874.272426] LustreError: 376198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11874.285085] LustreError: 376198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11877.592299] LustreError: 376184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452595 with bad export cookie 11306854208319847981 [11877.595918] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11877.614972] LustreError: 376184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11877.976271] Lustre: server umount lustre-MDT0001 complete [11887.785952] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11890.424441] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:23:25 (1788452605) [11898.849408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11899.218176] LustreError: 380624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11899.294712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11904.436974] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11904.481107] LustreError: 380625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.602797] LustreError: 380624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11914.721706] LustreError: 380625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.034363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11915.627610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11921.425121] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11930.074625] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11937.332343] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11946.420048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11947.081824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11954.053181] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11965.607624] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11974.268418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11974.474638] Lustre: lustre-OST0001: new disk, initializing [11974.487942] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11974.494305] Lustre: 383462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11974.589612] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11982.475610] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [11984.594518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11984.605402] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11984.660127] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11995.331841] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12003.137783] Lustre: lustre-OST0001: Not available for connect from 192.168.204.3@tcp (stopping) [12005.362726] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12005.367826] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12005.381484] Lustre: Skipped 1 previous similar message [12005.396575] LustreError: Skipped 1 previous similar message [12007.107568] Lustre: server umount lustre-OST0001 complete [12008.233054] LustreError: 382448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12025.824188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12025.826885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12025.827763] LustreError: 382447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12025.827771] LustreError: 382447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12025.838243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12025.855401] Lustre: Skipped 1 previous similar message [12025.905435] Lustre: Skipped 3 previous similar messages [12031.547920] Lustre: server umount lustre-OST0000 complete [12036.069243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12036.090991] Lustre: Skipped 1 previous similar message [12046.308404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.319755] Lustre: Skipped 7 previous similar messages [12050.400153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12050.718722] Lustre: server umount lustre-MDT0000 complete [12055.038233] LustreError: 380606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452772 with bad export cookie 11306854208319848856 [12055.040419] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12055.052992] LustreError: 380606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12061.519054] Lustre: server umount lustre-MDT0001 complete [12074.582709] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [12078.269593] Key type lgssc unregistered [12078.618681] LNet: 385455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12078.624391] LNetError: 385455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12078.656440] LNet: Removed LNI 192.168.204.103@tcp [12079.651427] Key type .llcrypt unregistered [12079.653827] Key type ._llcrypt unregistered [12139.356561] Key type ._llcrypt registered [12139.363523] Key type .llcrypt registered [12139.496064] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [12140.831233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12140.865800] alg: No test for adler32 (adler32-zlib) [12141.868036] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12141.996975] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [12143.824665] Key type lgssc registered [12145.077884] Lustre: Echo OBD driver; http://www.lustre.org/ [12153.605337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12153.654326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12155.249454] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12155.299923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12155.456094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12160.031717] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12170.172895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12170.434978] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12170.534056] Lustre: 388254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12170.983751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12176.672045] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12185.701411] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12193.458574] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12196.834294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12196.838659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12196.846986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12198.369959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12198.373137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12201.961313] Lustre: server umount lustre-MDT0000 complete [12206.221977] LustreError: 388247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452923 with bad export cookie 17630846778893663971 [12206.252194] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12206.775244] Lustre: server umount lustre-MDT0001 complete [12217.506552] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:28:52 (1788452932) [12225.116884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12225.560558] LustreError: 390767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12225.583447] LustreError: 390767:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12225.639106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12230.357725] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12230.631182] LustreError: 390768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12235.751119] LustreError: 390767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12239.202089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12239.525358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12243.883598] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12251.710510] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12257.652459] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12265.182487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12265.343959] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12265.547120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12272.771432] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12281.103754] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12289.301681] Lustre: Permanently deactivating lustre-OST0000 [12289.333245] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12295.189221] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12297.008275] LustreError: 393707:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12297.019176] LustreError: 393707:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12297.025466] LustreError: 393707:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12299.938910] LustreError: 393756:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12299.950791] LustreError: 393756:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12307.985621] Lustre: server umount lustre-OST0000 complete [12310.719389] LustreError: 394005:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12310.728996] LustreError: 394005:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12313.060552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12313.064444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12313.068467] Lustre: Skipped 1 previous similar message [12313.079192] Lustre: Skipped 2 previous similar messages [12318.176772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12318.187772] Lustre: Skipped 1 previous similar message [12322.450130] Lustre: server umount lustre-MDT0000 complete [12323.299196] LustreError: 390768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12323.334499] LustreError: 390768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12326.744292] LustreError: 393460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453044 with bad export cookie 17630846778893664545 [12326.748481] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12326.750200] LustreError: 393460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12337.277585] Lustre: server umount lustre-MDT0001 complete [12352.400055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12361.711435] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12364.966424] LustreError: 395166:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12372.498375] Lustre: 395506:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.103@tcp [12375.459991] Lustre: server umount MGS complete [12387.917114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12388.398807] LustreError: 396230:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12388.443776] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12388.450712] Lustre: Skipped 1 previous similar message [12388.537461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12394.130952] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12399.077607] LustreError: 396231:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12399.108987] LustreError: 396231:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12404.912571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12405.340568] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12405.462529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12410.473757] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12419.000539] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12425.308335] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12427.480180] Lustre: Permanently reactivating lustre-OST0000 [12427.494649] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12427.499141] Lustre: Skipped 2 previous similar messages [12453.346284] LustreError: 397938:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12463.841680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12464.454218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12473.635025] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12484.017995] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12494.304486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12494.316394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12494.328050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12494.335719] Lustre: Skipped 1 previous similar message [12496.668235] Lustre: server umount lustre-OST0000 complete [12502.497854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12502.504075] LustreError: Skipped 1 previous similar message [12502.507992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12502.521188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12502.530058] Lustre: Skipped 1 previous similar message [12507.073284] Lustre: server umount lustre-MDT0000 complete [12509.665566] LustreError: 398236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12509.704565] LustreError: 398236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12512.033850] LustreError: 399700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453229 with bad export cookie 17630846778893665595 [12512.044291] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12512.067949] LustreError: 399700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12512.773647] Lustre: server umount lustre-MDT0001 complete [12525.794857] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [12529.493855] Key type lgssc unregistered [12530.062372] LNet: 400455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12530.076822] LNetError: 400455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12531.118597] LNet: Removed LNI 192.168.204.103@tcp [12532.190964] Key type .llcrypt unregistered [12532.193496] Key type ._llcrypt unregistered [12555.881321] Key type ._llcrypt registered [12555.885273] Key type .llcrypt registered [12556.045188] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [12574.418689] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [12575.683909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12575.715025] alg: No test for adler32 (adler32-zlib) [12577.038896] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12577.362811] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [12579.072294] Key type lgssc registered [12580.237672] Lustre: Echo OBD driver; http://www.lustre.org/ [12622.349265] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12622.393157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12623.768128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12623.823244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12623.978924] Lustre: lustre-MDT0000: new disk, initializing [12624.060494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12624.084421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12630.211508] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12642.676372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12642.841460] Lustre: 403621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12642.904500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12642.909775] Lustre: Skipped 1 previous similar message [12643.002397] Lustre: lustre-MDT0001: new disk, initializing [12643.102768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12643.152375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12643.165910] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12648.456130] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12659.624048] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12668.031675] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12677.451540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12677.820934] Lustre: lustre-OST0000: new disk, initializing [12677.825421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12677.833930] Lustre: 405589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12677.932379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12685.424348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12685.444107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12685.595500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12685.856708] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12697.211398] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12703.015398] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12703.202113] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12708.522155] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12708.776023] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12711.394153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12711.399053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12711.442398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12711.449664] Lustre: Skipped 1 previous similar message [12716.525956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12716.543729] Lustre: Skipped 1 previous similar message [12721.632216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12721.643442] Lustre: Skipped 1 previous similar message [12725.215429] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12725.374162] Lustre: server umount lustre-OST0000 complete [12731.873694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12731.882627] Lustre: Skipped 1 previous similar message [12731.888880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12731.892913] Lustre: Skipped 1 previous similar message [12736.749728] Lustre: server umount lustre-MDT0000 complete [12741.771166] LustreError: 403611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453459 with bad export cookie 16868641324469735166 [12741.781202] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12741.795885] LustreError: 403611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12742.262491] Lustre: server umount lustre-MDT0001 complete [12752.964771] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12755.294446] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:37:50 (1788453470) [12763.736786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12764.213098] LustreError: 408052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12764.250365] LustreError: 408052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12764.325615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12769.765752] LustreError: 408053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12770.198342] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12774.890203] LustreError: 408052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12780.003612] LustreError: 408053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12780.936763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12781.572120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12788.011504] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12797.320438] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12804.908934] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12814.294995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12814.763064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12823.069111] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12832.307950] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12834.757033] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12837.643584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12845.541343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12845.543541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12845.545944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12845.571408] Lustre: Skipped 2 previous similar messages [12849.572854] Lustre: server umount lustre-OST0000 complete [12855.777857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12855.787340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12855.787797] Lustre: Skipped 1 previous similar message [12855.803392] Lustre: Skipped 2 previous similar messages [12860.385766] Lustre: server umount lustre-MDT0000 complete [12860.911384] LustreError: 408049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12860.927806] LustreError: 408049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12864.430966] LustreError: 408034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453582 with bad export cookie 16868641324469736048 [12864.441079] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12864.447247] LustreError: 408034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12865.163787] Lustre: server umount lustre-MDT0001 complete [12877.900877] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [12881.375503] Key type lgssc unregistered [12881.800734] LNet: 411781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12881.804909] LNetError: 411781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12881.817422] LNet: Removed LNI 192.168.204.103@tcp [12882.792127] Key type .llcrypt unregistered [12882.795150] Key type ._llcrypt unregistered [12916.823597] Key type ._llcrypt registered [12916.826126] Key type .llcrypt registered [12916.923156] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [12917.178515] Key type .llcrypt unregistered [12917.184052] Key type ._llcrypt unregistered [12938.143823] Key type ._llcrypt registered [12938.148863] Key type .llcrypt registered [12938.302973] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [12939.393894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12939.467194] alg: No test for adler32 (adler32-zlib) [12940.937367] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12941.375389] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [12943.071298] Key type lgssc registered [12944.710962] Lustre: Echo OBD driver; http://www.lustre.org/ [12954.748236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12954.801979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12956.516590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12961.535189] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12966.881352] LustreError: 413999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12966.902779] LustreError: 413999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12971.179978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12971.571548] LustreError: 413998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12971.617918] LustreError: 413998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12971.620690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12976.613427] LustreError: 413999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12976.636284] LustreError: 413999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12979.717301] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [12981.745509] LustreError: 413998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12981.787653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12981.794034] LustreError: 413998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12986.868147] LustreError: 413999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12986.896257] LustreError: 413999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12989.218503] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12997.099914] LustreError: 414742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12997.133271] LustreError: 414742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12998.313362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12998.850860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13003.891500] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13014.500779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13014.511841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13014.525189] Lustre: Skipped 1 previous similar message [13016.097588] Lustre: server umount lustre-OST0000 complete [13020.298817] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13020.307945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13020.322473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13020.326414] Lustre: Skipped 1 previous similar message [13024.737986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13024.739046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13026.564836] Lustre: server umount lustre-MDT0000 complete [13029.856662] LustreError: 413999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13029.879935] LustreError: 413999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13030.557057] LustreError: 413978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453748 with bad export cookie 12336813491088423551 [13030.558231] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13030.568496] LustreError: 413978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13031.086085] Lustre: server umount lustre-MDT0001 complete [13042.815825] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [13046.059585] Key type lgssc unregistered [13046.409704] LNet: 417153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13046.428849] LNetError: 417153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13046.448515] LNet: Removed LNI 192.168.204.103@tcp [13047.515148] Key type .llcrypt unregistered [13047.517315] Key type ._llcrypt unregistered [13072.767920] Key type ._llcrypt registered [13072.770822] Key type .llcrypt registered [13072.950919] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [13074.446983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13074.499267] alg: No test for adler32 (adler32-zlib) [13075.577545] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13075.929871] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [13077.655183] Key type lgssc registered [13078.889201] Lustre: Echo OBD driver; http://www.lustre.org/ [13086.262225] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13086.275981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13101.855485] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13106.975358] LustreError: 418545:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.103@tcp: failed processing log, type 4: rc = -110 [13133.665955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13140.883940] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13148.644916] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13156.081420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13156.674775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13156.765209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13161.466895] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13161.960774] LustreError: 418571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13161.981102] LustreError: 418571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13167.073384] LustreError: 418570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13169.125617] LustreError: 418571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13169.139953] LustreError: 418571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13172.053443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13172.447555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13177.767865] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13185.750476] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13192.646699] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13202.912648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13202.924682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13202.948669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13203.450508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13203.461233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13203.490122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13203.498067] Lustre: Skipped 1 previous similar message [13208.555306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13213.664373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13213.675229] Lustre: Skipped 1 previous similar message [13215.199302] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13215.504770] Lustre: server umount lustre-OST0000 complete [13223.910561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13223.912638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13223.925228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13223.925238] Lustre: Skipped 1 previous similar message [13223.933790] Lustre: Skipped 1 previous similar message [13226.667347] Lustre: server umount lustre-MDT0000 complete [13229.026534] LustreError: 421297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13229.047442] LustreError: 421297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13230.740597] LustreError: 418553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453948 with bad export cookie 16097912373321824148 [13230.742848] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13230.751527] LustreError: 418553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13231.195269] Lustre: server umount lustre-MDT0001 complete [13244.842792] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [13248.553443] Key type lgssc unregistered [13248.924766] LNet: 422121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13248.947453] LNetError: 422121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13250.021579] LNet: Removed LNI 192.168.204.103@tcp [13251.314357] Key type .llcrypt unregistered [13251.316406] Key type ._llcrypt unregistered [13281.675330] Key type ._llcrypt registered [13281.682946] Key type .llcrypt registered [13281.894611] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [13283.666252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13283.873848] alg: No test for adler32 (adler32-zlib) [13285.644477] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13286.298578] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [13288.143237] Key type lgssc registered [13290.181103] Lustre: Echo OBD driver; http://www.lustre.org/ [13300.129631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13300.162218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13301.855583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13307.347611] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13311.970952] LustreError: 423554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13312.001988] LustreError: 423554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13317.091501] LustreError: 423555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13317.914418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13318.442763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13323.574532] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13332.537166] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13341.294300] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13350.354174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13350.972726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13359.335943] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13370.535165] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13372.476809] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13380.064703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13380.070104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13380.089161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13381.942190] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.3@tcp (stopping) [13382.648937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13382.659762] Lustre: Skipped 1 previous similar message [13386.079839] Lustre: server umount lustre-MDT0000 complete [13387.032582] LustreError: 424541:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13388.988051] LustreError: 423536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.3@tcp arrived at 1788454106 with bad export cookie 16864093305993053540 [13391.202041] LustreError: 423535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454108 with bad export cookie 16864093305993052875 [13391.226215] LustreError: 423535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13391.230139] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13392.864260] LustreError: 423555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13392.869233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13392.882960] LustreError: 423555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13392.914120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13392.918520] Lustre: Skipped 3 previous similar messages [13396.896507] LustreError: 423554:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13396.899680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13396.920750] LustreError: 423554:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13397.529646] Lustre: server umount lustre-OST0000 complete [13405.013675] Lustre: server umount lustre-MDT0001 complete [13418.883482] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [13421.718310] Key type lgssc unregistered [13422.115986] LNet: 427247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13422.120960] LNetError: 427247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13422.137193] LNet: Removed LNI 192.168.204.103@tcp [13423.448838] Key type .llcrypt unregistered [13423.450610] Key type ._llcrypt unregistered [13453.836854] Key type ._llcrypt registered [13453.842128] Key type .llcrypt registered [13453.983029] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [13455.564180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13455.891113] alg: No test for adler32 (adler32-zlib) [13457.018064] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13457.358882] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [13459.079265] Key type lgssc registered [13460.649115] Lustre: Echo OBD driver; http://www.lustre.org/ [13470.656309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13470.689417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13472.310144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13478.091909] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13482.471303] LustreError: 428662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13482.482972] LustreError: 428662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13487.585895] LustreError: 428663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13487.723656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13488.192983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13493.377191] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13501.462251] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13507.357785] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13516.443111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13517.143269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13518.204953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13524.752853] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13535.292772] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13542.886256] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13542.905886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13542.921768] Lustre: Skipped 1 previous similar message [13546.892378] Lustre: server umount lustre-MDT0001 complete [13548.005846] LustreError: 428663:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13548.025339] LustreError: 428663:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13551.015616] LustreError: 431107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13551.048409] LustreError: 431107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13554.144554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13554.163917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13554.182462] Lustre: Skipped 1 previous similar message [13554.189595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13554.202862] Lustre: Skipped 1 previous similar message [13556.194149] LustreError: 428662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13556.210887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13560.460378] Lustre: server umount lustre-OST0000 complete [13565.347810] Lustre: server umount lustre-MDT0000 complete [13581.117588] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [13584.849622] Key type lgssc unregistered [13585.208170] LNet: 432353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13585.225317] LNetError: 432353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13586.281631] LNet: Removed LNI 192.168.204.103@tcp [13587.340814] Key type .llcrypt unregistered [13587.343860] Key type ._llcrypt unregistered [13616.966843] Key type ._llcrypt registered [13616.970716] Key type .llcrypt registered [13617.151649] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [13632.123940] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [13633.287966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13633.432393] alg: No test for adler32 (adler32-zlib) [13634.526236] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13634.747537] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [13636.431222] Key type lgssc registered [13637.734213] Lustre: Echo OBD driver; http://www.lustre.org/ [13680.140786] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [13698.711552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13698.746366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13700.099723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13700.129670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13700.246620] Lustre: lustre-MDT0000: new disk, initializing [13700.352342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13700.384217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13705.879284] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13723.847632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13724.150434] Lustre: lustre-OST0000: new disk, initializing [13724.154647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13724.158665] Lustre: Skipped 1 previous similar message [13724.166493] Lustre: 437390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13724.295037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13731.508383] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13732.474526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13732.492776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13732.559956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13742.265318] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13758.325616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13758.481667] Lustre: 436371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13758.552110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13758.627758] Lustre: lustre-MDT0001: new disk, initializing [13758.750603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13758.798594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13758.811972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13764.579782] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13777.525345] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13777.801318] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13783.689190] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13783.977093] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13793.082857] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.3@tcp (stopping) [13793.765499] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13793.767805] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13793.774857] Lustre: Skipped 1 previous similar message [13793.775625] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13795.928218] Lustre: server umount lustre-MDT0001 complete [13804.012656] LustreError: 439086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13804.043544] LustreError: 439086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13807.009714] LustreError: 436379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13807.055946] LustreError: 436379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13812.200653] LustreError: 439086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13812.241075] LustreError: 439086:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13817.316072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13817.322776] LustreError: 436383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13817.356845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13817.397115] Lustre: Skipped 1 previous similar message [13822.433119] LustreError: 437374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13822.572575] Lustre: server umount lustre-OST0000 complete [13833.035509] Lustre: server umount lustre-MDT0000 complete [13848.869196] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [13851.500683] Key type lgssc unregistered [13851.785942] LNet: 441143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13851.799981] LNetError: 441143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13851.819960] LNet: Removed LNI 192.168.204.103@tcp [13852.730611] Key type .llcrypt unregistered [13852.733504] Key type ._llcrypt unregistered [13904.101365] Key type ._llcrypt registered [13904.106353] Key type .llcrypt registered [13904.260762] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [13904.688698] Key type .llcrypt unregistered [13904.692758] Key type ._llcrypt unregistered [13923.942185] Key type ._llcrypt registered [13923.949896] Key type .llcrypt registered [13924.132404] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [13926.031205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13926.114767] alg: No test for adler32 (adler32-zlib) [13927.349696] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13927.668770] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [13929.367532] Key type lgssc registered [13931.438344] Lustre: Echo OBD driver; http://www.lustre.org/ [13966.873385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13966.921022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13968.330717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13968.383956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13968.556815] Lustre: lustre-MDT0000: new disk, initializing [13968.718751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13968.765396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13974.661300] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [13987.595495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13987.718735] Lustre: 444525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13987.790638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13987.793753] Lustre: Skipped 1 previous similar message [13987.843191] Lustre: lustre-MDT0001: new disk, initializing [13987.914146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13987.972200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13987.991410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13992.772201] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [14004.207924] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14012.627534] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14020.847569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14021.282226] Lustre: lustre-OST0000: new disk, initializing [14021.288731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14021.296397] Lustre: 446500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14021.402104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14027.839092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14027.853169] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14028.012509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14029.279947] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [14040.378213] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14048.739487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14048.741262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14048.776962] Lustre: Skipped 2 previous similar messages [14048.797594] Lustre: Skipped 2 previous similar messages [14053.862603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14053.877716] Lustre: Skipped 2 previous similar messages [14054.192911] Lustre: server umount lustre-MDT0000 complete [14058.945806] LustreError: 446499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454776 with bad export cookie 12971222768807043965 [14058.957983] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14058.969646] LustreError: 446499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14059.015943] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14059.049923] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14059.726511] Lustre: server umount lustre-MDT0001 complete [14067.246049] Lustre: server umount lustre-OST0000 complete [14082.230560] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:59:56 (1788454796) [14091.940232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14092.463937] LustreError: 448529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14092.481878] LustreError: 448529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14092.540605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14097.729259] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [14097.894643] LustreError: 448530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14103.012972] LustreError: 448530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14107.930948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14108.213128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14113.215621] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [14123.555331] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14132.444873] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14155.692885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14155.860911] Lustre: Found index 0 for lustre-OST0000, updating log [14156.043548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14165.232846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14165.618902] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [14176.491229] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14184.930748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14184.934409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14184.950493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14184.961170] Lustre: Skipped 1 previous similar message [14188.113409] Lustre: server umount lustre-OST0000 complete [14205.412783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14205.427994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14205.455134] Lustre: Skipped 1 previous similar message [14205.460567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14205.463043] Lustre: Skipped 2 previous similar messages [14209.777775] Lustre: server umount lustre-MDT0000 complete [14211.553624] LustreError: 449515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14211.579741] LustreError: 449515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14214.554322] LustreError: 448509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454932 with bad export cookie 12971222768807045351 [14214.574945] LustreError: 448509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14214.603512] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14215.233449] Lustre: server umount lustre-MDT0001 complete [14215.457674] LNet: 1 local NIs in recovery (showing 1): 192.168.204.103@tcp [14225.469928] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:02:20 (1788454940) [14863.048426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14863.390841] LustreError: 454954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14863.411845] LustreError: 454954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14863.458386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14868.458552] LustreError: 454955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14868.478510] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [14873.570512] LustreError: 454954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14877.847502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14878.357264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14882.623396] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [14890.652604] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14897.795046] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14919.177950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14919.562389] Lustre: Found index 0 for lustre-OST0000, updating log [14919.872934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14920.031277] LNet: 1 local NIs in recovery (showing 1): 192.168.204.103@tcp [14926.984059] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [14936.493751] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14940.846955] Lustre: server umount lustre-OST0000 complete [14947.299495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14947.313664] Lustre: Skipped 1 previous similar message [14947.319303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14947.327431] Lustre: Skipped 3 previous similar messages [14948.385034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14948.394206] Lustre: Skipped 1 previous similar message [14950.434488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14950.444493] Lustre: Skipped 3 previous similar messages [14952.889643] Lustre: server umount lustre-MDT0000 complete [14953.505547] LustreError: 455682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14953.541460] LustreError: 455682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14957.364498] LustreError: 454936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455675 with bad export cookie 12971222768807046436 [14957.372036] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14957.374115] LustreError: 454936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [14957.778208] Lustre: server umount lustre-MDT0001 complete [14979.423327] LNet: 1 local NIs in recovery (showing 1): 192.168.204.103@tcp [14988.760344] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [14999.486846] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [15042.623511] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:15:57 (1788455757) [15060.005491] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [15062.443845] Key type lgssc unregistered [15062.682896] LNet: 461575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15062.690495] LNetError: 461575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15181.482952] LNet: Removed LNI 192.168.204.103@tcp [15182.361899] Key type .llcrypt unregistered [15182.366155] Key type ._llcrypt unregistered [15198.097046] Key type ._llcrypt registered [15198.101717] Key type .llcrypt registered [15198.174959] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [15199.154663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15199.222199] alg: No test for adler32 (adler32-zlib) [15200.353515] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15200.666915] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [15202.367734] Key type lgssc registered [15203.468693] Lustre: Echo OBD driver; http://www.lustre.org/ [15207.328655] LNet: Added LNI 192.168.204.103@tcp100 [8/256/0/180] [15207.994448] LNet: Added LNI 192.168.204.103@tcp101 [8/256/0/180] [15208.691372] LNet: Added LNI 192.168.204.103@tcp102 [8/256/0/180] [15209.373530] LNet: Added LNI 192.168.204.103@tcp103 [8/256/0/180] [15210.176631] LNet: Added LNI 192.168.204.103@tcp104 [8/256/0/180] [15210.958928] LNet: Added LNI 192.168.204.103@tcp105 [8/256/0/180] [15211.792759] LNet: Added LNI 192.168.204.103@tcp106 [8/256/0/180] [15212.681343] LNet: Added LNI 192.168.204.103@tcp107 [8/256/0/180] [15213.829941] LNet: Added LNI 192.168.204.103@tcp108 [8/256/0/180] [15214.981371] LNet: Added LNI 192.168.204.103@tcp109 [8/256/0/180] [15215.807991] LNet: Added LNI 192.168.204.103@tcp110 [8/256/0/180] [15216.751192] LNet: Added LNI 192.168.204.103@tcp111 [8/256/0/180] [15217.376930] LNet: Added LNI 192.168.204.103@tcp112 [8/256/0/180] [15218.138982] LNet: Added LNI 192.168.204.103@tcp113 [8/256/0/180] [15218.969635] LNet: Added LNI 192.168.204.103@tcp114 [8/256/0/180] [15220.120086] LNet: Added LNI 192.168.204.103@tcp115 [8/256/0/180] [15220.992890] LNet: Added LNI 192.168.204.103@tcp116 [8/256/0/180] [15222.049612] LNet: Added LNI 192.168.204.103@tcp117 [8/256/0/180] [15223.035265] LNet: Added LNI 192.168.204.103@tcp118 [8/256/0/180] [15224.053742] LNet: Added LNI 192.168.204.103@tcp119 [8/256/0/180] [15224.838122] LNet: Added LNI 192.168.204.103@tcp120 [8/256/0/180] [15226.020502] LNet: Added LNI 192.168.204.103@tcp121 [8/256/0/180] [15226.714744] LNet: Added LNI 192.168.204.103@tcp122 [8/256/0/180] [15227.865604] LNet: Added LNI 192.168.204.103@tcp123 [8/256/0/180] [15228.816336] LNet: Added LNI 192.168.204.103@tcp124 [8/256/0/180] [15229.762377] LNet: Added LNI 192.168.204.103@tcp125 [8/256/0/180] [15230.696140] LNet: Added LNI 192.168.204.103@tcp126 [8/256/0/180] [15231.752824] LNet: Added LNI 192.168.204.103@tcp127 [8/256/0/180] [15232.721978] LNet: Added LNI 192.168.204.103@tcp128 [8/256/0/180] [15233.934743] LNet: Added LNI 192.168.204.103@tcp129 [8/256/0/180] [15234.938819] LNet: Added LNI 192.168.204.103@tcp130 [8/256/0/180] [15236.050435] LNet: Added LNI 192.168.204.103@tcp131 [8/256/0/180] [15237.088593] LNet: Added LNI 192.168.204.103@tcp132 [8/256/0/180] [15238.139531] LNet: Added LNI 192.168.204.103@tcp133 [8/256/0/180] [15239.135591] LNet: Added LNI 192.168.204.103@tcp134 [8/256/0/180] [15240.338942] LNet: Added LNI 192.168.204.103@tcp135 [8/256/0/180] [15248.784398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15248.834000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15250.003902] Lustre: 464607:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.103@tcp: can't add NID 192.168.204.103@tcp130: rc = -75 [15250.088105] LNetError: 464635:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.103 [15250.098844] LNetError: Protocol error connecting to 192.168.204.103@tcp117 at host 192.168.204.103:988: is it running a compatible version of Lustre? [15250.118873] LNetError: Skipped 4 previous similar messages [15250.182826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15250.207827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15250.274780] Lustre: lustre-MDT0000: new disk, initializing [15250.336642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15250.354182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15250.849543] LNet: 8 local NIs in recovery (showing 8): 192.168.204.103@tcp124, 192.168.204.103@tcp125, 192.168.204.103@tcp127, 192.168.204.103@tcp126, 192.168.204.103@tcp130, 192.168.204.103@tcp129, 192.168.204.103@tcp131, 192.168.204.103@tcp132 [15251.168921] LNetError: 464635:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.103 [15251.187845] LNetError: 464635:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [15251.209495] LNetError: Protocol error connecting to 192.168.204.103@tcp106 at host 192.168.204.103:988: is it running a compatible version of Lustre? [15251.224558] LNetError: Skipped 5 previous similar messages [15253.990418] LNetError: 464627:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.103 [15253.998446] LNetError: 464627:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [15254.023514] LNetError: Protocol error connecting to 192.168.204.103@tcp124 at host 192.168.204.103:988: is it running a compatible version of Lustre? [15254.042815] LNetError: Skipped 10 previous similar messages [15256.021674] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [15265.280289] LNetError: 462065:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.103 [15265.285203] LNetError: 462065:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [15265.297785] LNetError: Protocol error connecting to 192.168.204.103@tcp124 at host 192.168.204.103:988: is it running a compatible version of Lustre? [15265.313611] LNetError: Skipped 6 previous similar messages [15271.636316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15271.785432] Lustre: 464624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15271.875962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15271.880726] Lustre: Skipped 1 previous similar message [15271.946037] Lustre: lustre-MDT0001: new disk, initializing [15271.970040] LNetError: 464633:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.103 [15271.977789] LNetError: 464633:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15271.983965] LNetError: Protocol error connecting to 192.168.204.103@tcp133 at host 192.168.204.103:988: is it running a compatible version of Lustre? [15271.987990] Lustre: 465428:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.103@tcp: can't add NID 192.168.204.103@tcp130: rc = -75 [15272.006614] Lustre: 465428:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15272.087604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15272.124478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15272.134508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15278.221598] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [15280.216479] Lustre: 465776:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.103@tcp: can't add NID 192.168.204.103@tcp130: rc = -75 [15280.231286] Lustre: 465776:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15281.702495] LNetError: 464628:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.103 [15281.720868] LNetError: 464628:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15281.734302] LNetError: Protocol error connecting to 192.168.204.103@tcp124 at host 192.168.204.103:988: is it running a compatible version of Lustre? [15281.757532] LNetError: Skipped 10 previous similar messages [15290.507612] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15298.030375] LNetError: 464630:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.103 [15298.060559] LNetError: 464630:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [15298.096403] LNetError: Protocol error connecting to 192.168.204.103@tcp131 at host 192.168.204.103:988: is it running a compatible version of Lustre? [15298.110563] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15298.134915] LNetError: Skipped 16 previous similar messages [15306.425761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15306.957957] Lustre: lustre-OST0000: new disk, initializing [15306.963335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15306.972179] Lustre: 466602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15307.182974] Lustre: 466607:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.103@tcp: can't add NID 192.168.204.103@tcp130: rc = -75 [15307.195453] Lustre: 466607:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15307.309558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15310.367095] LNet: 3 local NIs in recovery (showing 3): 192.168.204.103@tcp135, 192.168.204.103@tcp133, 192.168.204.103@tcp124 [15316.034060] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [15317.090985] Lustre: 467008:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.103@tcp: can't add NID 192.168.204.103@tcp130: rc = -75 [15317.101406] Lustre: 467008:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15317.166564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15317.185170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15317.528017] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15326.125634] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15330.784682] LNetError: 464630:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.103 [15330.803552] LNetError: 464630:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 30 previous similar messages [15330.828039] LNetError: Protocol error connecting to 192.168.204.103@tcp125 at host 192.168.204.103:988: is it running a compatible version of Lustre? [15330.856704] LNetError: Skipped 30 previous similar messages [15337.955031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15337.970392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15337.979800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15337.997539] Lustre: Skipped 1 previous similar message [15338.034997] Lustre: Skipped 1 previous similar message [15339.346467] Lustre: server umount lustre-OST0000 complete [15343.075686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15343.075686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15343.076261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15343.098809] Lustre: Skipped 1 previous similar message [15348.198398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15348.212166] Lustre: Skipped 1 previous similar message [15348.823368] Lustre: server umount lustre-MDT0000 complete [15352.980297] LustreError: 465437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456070 with bad export cookie 723664476112017334 [15352.992141] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15352.997162] LustreError: 465437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15353.355828] Lustre: server umount lustre-MDT0001 complete [15366.033259] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [15369.292113] Key type lgssc unregistered [15369.806278] LNet: 468521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15369.818654] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15369.829487] LNet: Removed LNI 192.168.204.103@tcp [15370.852178] LNet: Removed LNI 192.168.204.103@tcp100 [15370.858809] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15370.864816] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15370.869557] LNet: Removed LNI 192.168.204.103@tcp101 [15371.935156] LNet: Removed LNI 192.168.204.103@tcp102 [15371.938797] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15371.957168] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15371.967597] LNet: Removed LNI 192.168.204.103@tcp103 [15373.023206] LNet: Removed LNI 192.168.204.103@tcp104 [15374.047197] LNet: Removed LNI 192.168.204.103@tcp105 [15374.051494] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15374.069407] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15374.087247] LNet: Removed LNI 192.168.204.103@tcp106 [15375.135291] LNet: Removed LNI 192.168.204.103@tcp107 [15376.159330] LNet: Removed LNI 192.168.204.103@tcp108 [15377.186059] LNet: Removed LNI 192.168.204.103@tcp109 [15378.271326] LNet: Removed LNI 192.168.204.103@tcp110 [15378.304244] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15378.318904] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15378.332842] LNet: Removed LNI 192.168.204.103@tcp111 [15379.359290] LNet: Removed LNI 192.168.204.103@tcp112 [15380.383402] LNet: Removed LNI 192.168.204.103@tcp113 [15380.393052] LNet: Removed LNI 192.168.204.103@tcp114 [15381.407880] LNet: Removed LNI 192.168.204.103@tcp115 [15382.431169] LNet: Removed LNI 192.168.204.103@tcp116 [15383.458575] LNet: Removed LNI 192.168.204.103@tcp117 [15384.480612] LNet: Removed LNI 192.168.204.103@tcp118 [15385.503152] LNet: Removed LNI 192.168.204.103@tcp119 [15386.527301] LNet: Removed LNI 192.168.204.103@tcp120 [15386.544201] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15386.590812] LNetError: 468521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [15386.609948] LNet: Removed LNI 192.168.204.103@tcp121 [15387.679748] LNet: Removed LNI 192.168.204.103@tcp122 [15388.704565] LNet: Removed LNI 192.168.204.103@tcp123 [15388.706666] LNet: Removed LNI 192.168.204.103@tcp124 [15389.727416] LNet: Removed LNI 192.168.204.103@tcp125 [15390.751367] LNet: Removed LNI 192.168.204.103@tcp126 [15391.775273] LNet: Removed LNI 192.168.204.103@tcp127 [15392.799549] LNet: Removed LNI 192.168.204.103@tcp128 [15393.824246] LNet: Removed LNI 192.168.204.103@tcp129 [15394.847594] LNet: Removed LNI 192.168.204.103@tcp130 [15395.873783] LNet: Removed LNI 192.168.204.103@tcp131 [15396.897801] LNet: Removed LNI 192.168.204.103@tcp132 [15396.907818] LNet: Removed LNI 192.168.204.103@tcp133 [15397.983546] LNet: Removed LNI 192.168.204.103@tcp134 [15398.030140] LNet: Removed LNI 192.168.204.103@tcp135 [15399.427899] Key type .llcrypt unregistered [15399.431612] Key type ._llcrypt unregistered [15425.388312] Key type ._llcrypt registered [15425.390669] Key type .llcrypt registered [15425.596642] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [15444.895916] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [15446.700749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15446.749836] alg: No test for adler32 (adler32-zlib) [15447.991628] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15448.393719] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [15450.167196] Key type lgssc registered [15451.593963] Lustre: Echo OBD driver; http://www.lustre.org/ [15495.274420] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:23:30 (1788456210) [15514.896182] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [15518.067240] Key type lgssc unregistered [15518.369067] LNet: 472615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15518.377692] LNetError: 472615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15518.389503] LNet: Removed LNI 192.168.204.103@tcp [15519.290643] Key type .llcrypt unregistered [15519.295061] Key type ._llcrypt unregistered [15539.131272] Key type ._llcrypt registered [15539.134351] Key type .llcrypt registered [15539.271370] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [15540.995414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15541.029111] alg: No test for adler32 (adler32-zlib) [15542.085559] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15542.326877] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [15544.007243] Key type lgssc registered [15545.247414] Lustre: Echo OBD driver; http://www.lustre.org/ [15556.614447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15556.640326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15557.895447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15557.931332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15558.037427] Lustre: lustre-MDT0000: new disk, initializing [15558.151821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15558.177925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15562.705465] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [15576.780455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15576.967731] Lustre: 473828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15577.070531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15577.078193] Lustre: Skipped 1 previous similar message [15577.222287] Lustre: lustre-MDT0001: new disk, initializing [15577.351447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15577.422260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15577.453447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15583.089222] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [15593.889695] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15600.985144] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15609.722149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15609.931562] Lustre: lustre-OST0000: new disk, initializing [15609.935982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15609.942594] Lustre: 475800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15610.043406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15617.156795] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [15620.182674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15620.208119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15620.385543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15627.627585] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15632.558693] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing lnet_if_list [15640.564720] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15640.615378] LNet: Added LNI 192.168.204.103@tcp10 [8/256/0/180] [15650.803094] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15650.829758] LNet: Added LNI 192.168.204.103@tcp11 [8/256/0/180] [15656.581920] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15656.611966] LNet: Added LNI 192.168.204.103@tcp12 [8/256/0/180] [15662.517940] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15662.574398] LNet: Added LNI 192.168.204.103@tcp13 [8/256/0/180] [15669.016213] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15669.064674] LNet: Added LNI 192.168.204.103@tcp14 [8/256/0/180] [15675.777434] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15675.801914] LNet: Added LNI 192.168.204.103@tcp15 [8/256/0/180] [15682.012412] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15682.098748] LNet: Added LNI 192.168.204.103@tcp16 [8/256/0/180] [15688.205272] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15688.248681] LNet: Added LNI 192.168.204.103@tcp17 [8/256/0/180] [15694.808696] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15694.865800] LNet: Added LNI 192.168.204.103@tcp18 [8/256/0/180] [15702.082160] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15702.118903] LNet: Added LNI 192.168.204.103@tcp19 [8/256/0/180] [15709.084200] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15709.151240] LNet: Added LNI 192.168.204.103@tcp20 [8/256/0/180] [15715.992798] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15716.038374] LNet: Added LNI 192.168.204.103@tcp21 [8/256/0/180] [15723.323891] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15723.360381] LNet: Added LNI 192.168.204.103@tcp22 [8/256/0/180] [15729.849272] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15729.883902] LNet: Added LNI 192.168.204.103@tcp23 [8/256/0/180] [15736.420774] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15736.473742] LNet: Added LNI 192.168.204.103@tcp24 [8/256/0/180] [15743.348858] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15743.377809] LNet: Added LNI 192.168.204.103@tcp25 [8/256/0/180] [15750.409523] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15750.431520] LNet: Added LNI 192.168.204.103@tcp26 [8/256/0/180] [15756.272858] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15756.301905] LNet: Added LNI 192.168.204.103@tcp27 [8/256/0/180] [15762.376380] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15762.421355] LNet: Added LNI 192.168.204.103@tcp28 [8/256/0/180] [15771.789853] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15771.833138] LNet: Added LNI 192.168.204.103@tcp29 [8/256/0/180] [15779.524192] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15779.568664] LNet: Added LNI 192.168.204.103@tcp30 [8/256/0/180] [15789.606677] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15789.654690] LNet: Added LNI 192.168.204.103@tcp31 [8/256/0/180] [15799.915330] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15799.959037] LNet: Added LNI 192.168.204.103@tcp32 [8/256/0/180] [15808.166248] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15808.224789] LNet: Added LNI 192.168.204.103@tcp33 [8/256/0/180] [15816.852237] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15816.890863] LNet: Added LNI 192.168.204.103@tcp34 [8/256/0/180] [15824.887536] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15824.975102] LNet: Added LNI 192.168.204.103@tcp35 [8/256/0/180] [15833.836989] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15833.886392] LNet: Added LNI 192.168.204.103@tcp36 [8/256/0/180] [15843.377055] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15843.408693] LNet: Added LNI 192.168.204.103@tcp37 [8/256/0/180] [15851.625501] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15851.678777] LNet: Added LNI 192.168.204.103@tcp38 [8/256/0/180] [15860.962565] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15861.020350] LNet: Added LNI 192.168.204.103@tcp39 [8/256/0/180] [15870.220706] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15870.264236] LNet: Added LNI 192.168.204.103@tcp40 [8/256/0/180] [15880.199117] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15880.301428] LNet: Added LNI 192.168.204.103@tcp41 [8/256/0/180] [15891.328246] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15891.370951] LNet: Added LNI 192.168.204.103@tcp42 [8/256/0/180] [15901.231963] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15901.273446] LNet: Added LNI 192.168.204.103@tcp43 [8/256/0/180] [15912.656537] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15912.679239] LNet: Added LNI 192.168.204.103@tcp44 [8/256/0/180] [15924.600172] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15924.643992] LNet: Added LNI 192.168.204.103@tcp45 [8/256/0/180] [15934.205692] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15934.269479] LNet: Added LNI 192.168.204.103@tcp46 [8/256/0/180] [15946.043846] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15946.098151] LNet: Added LNI 192.168.204.103@tcp47 [8/256/0/180] [15957.398964] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15957.439173] LNet: Added LNI 192.168.204.103@tcp48 [8/256/0/180] [15966.387781] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15966.466869] LNet: Added LNI 192.168.204.103@tcp49 [8/256/0/180] [15976.709296] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15976.758757] LNet: Added LNI 192.168.204.103@tcp50 [8/256/0/180] [16007.653369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16007.674137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16007.719294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16009.700693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16009.703200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16009.722983] Lustre: Skipped 1 previous similar message [16012.048464] Lustre: server umount lustre-MDT0000 complete [16016.256070] LustreError: 473820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456734 with bad export cookie 9750272780173282675 [16016.260745] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16016.268464] LustreError: 473820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16016.605985] Lustre: server umount lustre-MDT0001 complete [16035.295169] Lustre: 473103:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456737/real 1788456737] req@00000000135d99a4 x1875332311481472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16035.332115] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16037.407330] Lustre: 473103:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456739/real 1788456739] req@0000000077e58622 x1875332311481600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16038.890494] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [16040.424906] Lustre: 473101:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456742/real 1788456742] req@000000007ce7a0cc x1875332311481728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456758 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16042.469081] Lustre: 473100:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456744/real 1788456744] req@000000006733a7ef x1875332311481984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16047.584622] Lustre: 473103:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788456749/real 1788456749] req@00000000b308298d x1875332311482112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788456765 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16049.572781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16061.984034] LustreError: 473099:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000068e7de84 x1875332311482880/t0(0) o250->MGC192.168.204.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16062.604567] LustreError: 475793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16062.639573] LustreError: 475793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [16062.787936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16064.803734] LustreError: 475794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16064.811818] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16065.830730] LustreError: 475793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16069.180850] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16071.145227] LustreError: 475794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16076.256601] LustreError: 475793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16081.360611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16081.899672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16087.028917] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16087.957287] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16097.834839] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16106.346704] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16129.507935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16129.519611] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16129.534167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16129.538207] Lustre: Skipped 2 previous similar messages [16133.092221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16133.092254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16135.328491] Lustre: server umount lustre-OST0000 complete [16143.328928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16143.330066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16143.331484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16143.331490] Lustre: Skipped 1 previous similar message [16143.371756] Lustre: Skipped 2 previous similar messages [16147.552600] Lustre: server umount lustre-MDT0000 complete [16148.448368] LustreError: 500075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16148.484993] LustreError: 500075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16153.089400] LustreError: 474626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456870 with bad export cookie 9750272780173285237 [16153.105490] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16153.116038] LustreError: 474626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16153.495883] Lustre: server umount lustre-MDT0001 complete [16167.653862] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [16170.799323] Key type lgssc unregistered [16171.384921] LNet: 502882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16171.400440] LNetError: 502882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16171.412803] LNet: Removed LNI 192.168.204.103@tcp [16171.417360] LNet: Removed LNI 192.168.204.103@tcp10 [16171.420593] LNet: Removed LNI 192.168.204.103@tcp11 [16171.427213] LNet: Removed LNI 192.168.204.103@tcp12 [16171.430186] LNet: Removed LNI 192.168.204.103@tcp13 [16171.438663] LNet: Removed LNI 192.168.204.103@tcp14 [16171.460040] LNet: Removed LNI 192.168.204.103@tcp15 [16171.461857] LNet: Removed LNI 192.168.204.103@tcp16 [16171.472340] LNet: Removed LNI 192.168.204.103@tcp17 [16171.480711] LNet: Removed LNI 192.168.204.103@tcp18 [16171.486637] LNet: Removed LNI 192.168.204.103@tcp19 [16171.496697] LNet: Removed LNI 192.168.204.103@tcp20 [16171.505807] LNet: Removed LNI 192.168.204.103@tcp21 [16171.514488] LNet: Removed LNI 192.168.204.103@tcp22 [16171.518459] LNet: Removed LNI 192.168.204.103@tcp23 [16171.520662] LNet: Removed LNI 192.168.204.103@tcp24 [16171.523393] LNet: Removed LNI 192.168.204.103@tcp25 [16171.527688] LNet: Removed LNI 192.168.204.103@tcp26 [16171.530665] LNet: Removed LNI 192.168.204.103@tcp27 [16171.534514] LNet: Removed LNI 192.168.204.103@tcp28 [16171.537697] LNet: Removed LNI 192.168.204.103@tcp29 [16171.540371] LNet: Removed LNI 192.168.204.103@tcp30 [16171.542886] LNet: Removed LNI 192.168.204.103@tcp31 [16171.546777] LNet: Removed LNI 192.168.204.103@tcp32 [16171.549347] LNet: Removed LNI 192.168.204.103@tcp33 [16171.551345] LNet: Removed LNI 192.168.204.103@tcp34 [16171.553660] LNet: Removed LNI 192.168.204.103@tcp35 [16171.557833] LNet: Removed LNI 192.168.204.103@tcp36 [16171.560613] LNet: Removed LNI 192.168.204.103@tcp37 [16171.564730] LNet: Removed LNI 192.168.204.103@tcp38 [16171.573244] LNet: Removed LNI 192.168.204.103@tcp39 [16171.578289] LNet: Removed LNI 192.168.204.103@tcp40 [16171.583490] LNet: Removed LNI 192.168.204.103@tcp41 [16171.594242] LNet: Removed LNI 192.168.204.103@tcp42 [16171.606274] LNet: Removed LNI 192.168.204.103@tcp43 [16171.612231] LNet: Removed LNI 192.168.204.103@tcp44 [16171.618235] LNet: Removed LNI 192.168.204.103@tcp45 [16171.621117] LNet: Removed LNI 192.168.204.103@tcp46 [16171.626233] LNet: Removed LNI 192.168.204.103@tcp47 [16171.630849] LNet: Removed LNI 192.168.204.103@tcp48 [16171.639402] LNet: Removed LNI 192.168.204.103@tcp49 [16171.726571] LNet: Removed LNI 192.168.204.103@tcp50 [16172.746609] Key type .llcrypt unregistered [16172.753272] Key type ._llcrypt unregistered [16197.194354] Key type ._llcrypt registered [16197.202494] Key type .llcrypt registered [16197.298275] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [16216.388349] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [16217.599336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16217.632683] alg: No test for adler32 (adler32-zlib) [16218.673145] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [16218.887511] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [16220.527264] Key type lgssc registered [16221.650341] Lustre: Echo OBD driver; http://www.lustre.org/ [16266.118000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16266.186376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16267.559294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16267.594712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16267.694367] Lustre: lustre-MDT0000: new disk, initializing [16267.803156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16267.840551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16273.124904] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16285.641561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16285.805949] Lustre: 506085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16285.871802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16285.877436] Lustre: Skipped 1 previous similar message [16285.990211] Lustre: lustre-MDT0001: new disk, initializing [16286.134087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16286.273826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16286.288588] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16291.700927] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16303.004363] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16309.569073] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16318.599927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16318.818708] Lustre: *** cfs_fail_loc=90f, val=0*** [16318.886067] Lustre: lustre-OST0000: new disk, initializing [16318.891262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16318.898725] Lustre: 508100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16319.024021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16324.679279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16324.696092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16324.849135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16326.219177] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16337.896863] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16350.689229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16350.695166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16350.697394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16350.711555] Lustre: Skipped 1 previous similar message [16353.879707] Lustre: server umount lustre-OST0000 complete [16360.929033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16360.932107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16360.957114] Lustre: Skipped 1 previous similar message [16360.981701] Lustre: Skipped 2 previous similar messages [16365.069635] Lustre: server umount lustre-MDT0000 complete [16369.941849] LustreError: 506075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457087 with bad export cookie 10645694521616596508 [16369.942781] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16369.960485] LustreError: 506075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16370.472600] Lustre: server umount lustre-MDT0001 complete [16382.973772] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [16386.521471] Key type lgssc unregistered [16386.823310] LNet: 510010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16386.829431] LNetError: 510010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16387.878626] LNet: Removed LNI 192.168.204.103@tcp [16388.918714] Key type .llcrypt unregistered [16388.924170] Key type ._llcrypt unregistered [16411.709412] Key type ._llcrypt registered [16411.714823] Key type .llcrypt registered [16411.879254] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [16413.036964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16413.052274] alg: No test for adler32 (adler32-zlib) [16414.090733] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [16414.371230] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [16416.055216] Key type lgssc registered [16417.222089] Lustre: Echo OBD driver; http://www.lustre.org/ [16426.128254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16426.163963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16427.740974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16433.327635] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16438.265474] LustreError: 511280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16438.294319] LustreError: 511280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16443.366872] LustreError: 511279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16443.614495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16443.968163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16449.128743] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16458.698455] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16465.508076] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16473.406607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16473.639584] LustreError: 513102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16473.851751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16481.270384] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16490.629231] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16504.620544] loop0: detected capacity change from 0 to 204800000 [16504.914047] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16504.931560] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16504.943205] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16512.522392] loop0: detected capacity change from 0 to 204800000 [16512.847548] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16512.862240] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16512.879864] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16519.470627] loop0: detected capacity change from 0 to 204800000 [16525.560444] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16525.822282] Lustre: fs15246-OST0000: new disk, initializing [16525.831140] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16525.838604] Lustre: 514875:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16525.915228] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16533.126889] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16543.304767] loop1: detected capacity change from 0 to 204800000 [16549.116620] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16549.258560] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16549.307840] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16549.376346] Lustre: fs15246-MDT0000: new disk, initializing [16549.539190] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16549.605445] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16549.754071] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16550.817741] Lustre: 515999:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000972e99ee x1875333225693568/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788457284 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16554.906591] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16559.997890] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16560.018178] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16560.055788] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16584.174565] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16584.221396] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16585.537820] Lustre: server umount fs15246-MDT0000 complete [16589.301510] LustreError: 511275:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16594.402826] LustreError: 511275:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16596.336656] Lustre: server umount fs15246-OST0000 complete [16611.298462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16611.309616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16611.334716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16612.322818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16612.332388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16612.353288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16614.890812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16614.902973] Lustre: Skipped 1 previous similar message [16616.585026] Lustre: server umount lustre-OST0000 complete [16623.074103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16623.090933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16623.117243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16628.925442] Lustre: server umount lustre-MDT0000 complete [16630.246509] LustreError: 511274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16630.279748] LustreError: 511274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16633.557895] LustreError: 511259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457351 with bad export cookie 3206366386810404254 [16633.566396] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16633.573920] LustreError: 511259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16634.112638] Lustre: server umount lustre-MDT0001 complete [16647.257583] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [16651.324172] Key type lgssc unregistered [16651.857664] LNet: 519098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16651.870403] LNetError: 519098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16651.900190] LNet: Removed LNI 192.168.204.103@tcp [16653.192990] Key type .llcrypt unregistered [16653.195620] Key type ._llcrypt unregistered [16675.591163] Key type ._llcrypt registered [16675.592984] Key type .llcrypt registered [16675.730717] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [16676.719312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16676.762992] alg: No test for adler32 (adler32-zlib) [16678.256964] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [16678.548768] LNet: Added LNI 192.168.204.103@tcp [8/256/0/180] [16680.231311] Key type lgssc registered [16681.516572] Lustre: Echo OBD driver; http://www.lustre.org/ [16727.341367] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [16740.381241] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [16784.471778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16784.497759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16785.684863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16785.723668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16785.777204] Lustre: lustre-MDT0000: new disk, initializing [16785.848243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16785.869965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16790.237789] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16801.807819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16801.940233] Lustre: 523577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16801.996656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16802.008129] Lustre: Skipped 1 previous similar message [16802.097353] Lustre: lustre-MDT0001: new disk, initializing [16802.210300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16802.250532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16802.264433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16807.117230] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16818.483456] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16825.877582] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16833.732610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16834.141128] Lustre: lustre-OST0000: new disk, initializing [16834.147558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16834.156050] Lustre: 525550:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16834.258045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16840.694883] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_default_debug -1 all [16843.289577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16843.302653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16843.372404] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16852.603411] Lustre: DEBUG MARKER: oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16859.127503] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16859.425220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16865.085871] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16865.470225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16868.834112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16868.835035] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16868.849576] LustreError: Skipped 1 previous similar message [16868.850594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16868.888787] Lustre: Skipped 1 previous similar message [16873.925527] Lustre: server umount lustre-OST0000 complete [16879.081215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16879.082695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16879.084865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16879.104671] Lustre: Skipped 1 previous similar message [16879.119639] Lustre: Skipped 1 previous similar message [16884.192880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16884.209108] Lustre: Skipped 2 previous similar messages [16884.952912] Lustre: server umount lustre-MDT0000 complete [16889.567261] LustreError: 524375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457607 with bad export cookie 10138470247987394600 [16889.593354] LustreError: MGC192.168.204.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16889.596628] LustreError: 524375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16890.193965] Lustre: server umount lustre-MDT0001 complete [16918.247490] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing set_hostid [16930.185831] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing load_modules_local [16971.912435] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16693 sec ========= 13:48:06 (1788457686) [16974.235790] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:48:09 (1788457689) === [16979.695161] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:48:14 (1788457694) === [17035.339983] Lustre: DEBUG MARKER: oleg403-server.virtnet: executing unload_modules_local [17038.709882] Key type lgssc unregistered [17039.062729] LNet: 532205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17039.072957] LNetError: 532205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17039.127721] LNet: Removed LNI 192.168.204.103@tcp [17040.296795] Key type .llcrypt unregistered [17040.300162] Key type ._llcrypt unregistered