[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 591710650 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002227] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007247] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009020] pid_max: default: 32768 minimum: 301 [ 0.010106] LSM: Security Framework initializing [ 0.011039] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013070] *** VALIDATE selinux *** [ 0.021501] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025826] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027108] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029066] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] *** VALIDATE tmpfs *** [ 0.032331] *** VALIDATE proc *** [ 0.033250] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038040] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042393] debug: unmapping init [mem 0xffffffff8fa59000-0xffffffff8fa60fff] [ 0.045190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046743] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054302] rcu: Hierarchical SRCU implementation. [ 0.056466] smp: Bringing up secondary CPUs ... [ 0.057667] x86: Booting SMP configuration: [ 0.058024] .... node #0, CPUs: #1 #2 #3 [ 0.063090] smp: Brought up 1 node, 4 CPUs [ 0.065018] smpboot: Max logical packages: 1 [ 0.066016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142026] node 0 deferred pages initialised in 72ms [ 0.146013] devtmpfs: initialized [ 0.147272] x86/mm: Memory block size: 128MB [ 0.149315] gcov: version magic: 0x41383552 [ 0.152317] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158309] pinctrl core: initialized pinctrl subsystem [ 0.160255] [ 0.160962] ************************************************************* [ 0.164012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166011] ** ** [ 0.169010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171011] ** ** [ 0.173012] ** This means that this kernel is built to expose internal ** [ 0.176013] ** IOMMU data structures, which may compromise security on ** [ 0.178024] ** your system. ** [ 0.181013] ** ** [ 0.183011] ** If you see this message and you are not debugging the ** [ 0.185011] ** kernel, report this immediately to your vendor! ** [ 0.188009] ** ** [ 0.190011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192010] ************************************************************* [ 0.195748] NET: Registered protocol family 16 [ 0.197406] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207140] cpuidle: using governor menu [ 0.208872] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211344] PCI: Using configuration type 1 for base access [ 0.213128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.222893] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228084] cryptd: max_cpu_qlen set to 1000 [ 0.229356] ACPI: Added _OSI(Module Device) [ 0.231013] ACPI: Added _OSI(Processor Device) [ 0.232009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238364] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246141] ACPI: Interpreter enabled [ 0.247053] ACPI: PM: (supports S0 S3 S4 S5) [ 0.248013] ACPI: Using IOAPIC for interrupt routing [ 0.250089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.253408] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263858] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.266035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.269023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278502] acpiphp: Slot [2] registered [ 0.280122] acpiphp: Slot [5] registered [ 0.281126] acpiphp: Slot [6] registered [ 0.283150] acpiphp: Slot [7] registered [ 0.285114] acpiphp: Slot [8] registered [ 0.286101] acpiphp: Slot [9] registered [ 0.288117] acpiphp: Slot [10] registered [ 0.289145] acpiphp: Slot [3] registered [ 0.291099] acpiphp: Slot [4] registered [ 0.292157] acpiphp: Slot [11] registered [ 0.294089] acpiphp: Slot [12] registered [ 0.295093] acpiphp: Slot [13] registered [ 0.297102] acpiphp: Slot [14] registered [ 0.298153] acpiphp: Slot [15] registered [ 0.300095] acpiphp: Slot [16] registered [ 0.302154] acpiphp: Slot [17] registered [ 0.303120] acpiphp: Slot [18] registered [ 0.305095] acpiphp: Slot [19] registered [ 0.306093] acpiphp: Slot [20] registered [ 0.308110] acpiphp: Slot [21] registered [ 0.310103] acpiphp: Slot [22] registered [ 0.311100] acpiphp: Slot [23] registered [ 0.313155] acpiphp: Slot [24] registered [ 0.314212] acpiphp: Slot [25] registered [ 0.316184] acpiphp: Slot [26] registered [ 0.318103] acpiphp: Slot [27] registered [ 0.319162] acpiphp: Slot [28] registered [ 0.321112] acpiphp: Slot [29] registered [ 0.323110] acpiphp: Slot [30] registered [ 0.324123] acpiphp: Slot [31] registered [ 0.326066] PCI host bridge to bus 0000:00 [ 0.328019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.330025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.333024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.336013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.342026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.344543] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.347998] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351389] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.361013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.397057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.402017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.406019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.408556] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.410554] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.413048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.416884] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.422013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.436015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.442017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.451148] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.458017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.469019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.501017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.513886] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.523015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.536016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.556017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.568327] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.578015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.585024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.606032] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.619697] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.627016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.633016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.647022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.658739] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.663018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.670014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.689017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.702149] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.709016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.716019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.733025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.749324] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.751420] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.753375] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.756327] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.758313] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.763168] iommu: Default domain type: Passthrough [ 0.764804] SCSI subsystem initialized [ 0.767162] ACPI: bus type USB registered [ 0.769144] usbcore: registered new interface driver usbfs [ 0.771069] usbcore: registered new interface driver hub [ 0.773086] usbcore: registered new device driver usb [ 0.775179] pps_core: LinuxPPS API ver. 1 registered [ 0.778013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.782064] PTP clock support registered [ 0.784223] EDAC MC: Ver: 3.0.0 [ 0.785885] PCI: Using ACPI for IRQ routing [ 0.787082] NetLabel: Initializing [ 0.789010] NetLabel: domain hash size = 128 [ 0.791011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.793081] NetLabel: unlabeled traffic allowed by default [ 0.796413] vgaarb: loaded [ 0.798040] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.800015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.807316] clocksource: Switched to clocksource kvm-clock [ 0.929385] VFS: Disk quotas dquot_6.6.0 [ 0.931102] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.933785] *** VALIDATE ramfs *** [ 0.934506] *** VALIDATE hugetlbfs *** [ 0.936099] pnp: PnP ACPI init [ 0.937473] pnp: PnP ACPI: found 6 devices [ 0.954182] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.958191] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.960703] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.963285] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.965959] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.968852] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.971858] NET: Registered protocol family 2 [ 0.974528] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.979607] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.983703] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.989439] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.993080] TCP: Hash tables configured (established 65536 bind 65536) [ 0.996161] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.999450] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.003162] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.006368] NET: Registered protocol family 1 [ 1.008747] RPC: Registered named UNIX socket transport module. [ 1.011077] RPC: Registered udp transport module. [ 1.012553] RPC: Registered tcp transport module. [ 1.014206] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.016231] NET: Registered protocol family 44 [ 1.017504] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.018855] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.021457] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.024477] PCI: CLS 0 bytes, default 64 [ 1.026500] Unpacking initramfs... [ 2.458299] debug: unmapping init [mem 0xffff8e01fcc54000-0xffff8e01fffbffff] [ 2.462423] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.464718] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.467639] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.998192] Initialise system trusted keyrings [ 2.999867] Key type blacklist registered [ 3.001804] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.010506] zbud: loaded [ 3.013894] *** VALIDATE nfs *** [ 3.015527] *** VALIDATE nfs4 *** [ 3.017338] pstore: using deflate compression [ 3.022248] Platform Keyring initialized [ 3.132565] NET: Registered protocol family 38 [ 3.133904] Key type asymmetric registered [ 3.134815] Asymmetric key parser 'x509' registered [ 3.136099] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.138271] io scheduler mq-deadline registered [ 3.140249] io scheduler kyber registered [ 3.141873] io scheduler bfq registered [ 3.144302] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.147275] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.149730] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.152611] ACPI: Power Button [PWRF] [ 3.158145] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.164518] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.177312] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.183354] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.216102] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.246410] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.277546] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.282235] Non-volatile memory driver v1.3 [ 3.283803] Linux agpgart interface v0.103 [ 3.313377] virtio_blk virtio1: [vda] 146960 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.315565] vda: detected capacity change from 0 to 75243520 [ 3.334650] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.337522] vdb: detected capacity change from 0 to 1073741824 [ 3.352876] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.356295] vdc: detected capacity change from 0 to 2621440000 [ 3.372190] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.375209] vdd: detected capacity change from 0 to 2621440000 [ 3.389251] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.392328] vde: detected capacity change from 0 to 4294967296 [ 3.407526] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.409635] vdf: detected capacity change from 0 to 4294967296 [ 3.418708] libphy: Fixed MDIO Bus: probed [ 3.426253] usbcore: registered new interface driver usbserial_generic [ 3.429537] usbserial: USB Serial support registered for generic [ 3.432274] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.437841] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.440219] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.442551] mousedev: PS/2 mouse device common for all mice [ 3.445592] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.449267] rtc_cmos 00:05: RTC can wake from S4 [ 3.452883] rtc_cmos 00:05: registered as rtc0 [ 3.454129] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.454879] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.460571] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.461763] intel_pstate: CPU model not supported [ 3.467259] hid: raw HID events driver (C) Jiri Kosina [ 3.469557] usbcore: registered new interface driver usbhid [ 3.471659] usbhid: USB HID core driver [ 3.473372] drop_monitor: Initializing network drop monitor service [ 3.475924] Initializing XFRM netlink socket [ 3.478097] NET: Registered protocol family 10 [ 3.481172] Segment Routing with IPv6 [ 3.482602] NET: Registered protocol family 17 [ 3.484680] mpls_gso: MPLS GSO support [ 3.490126] RAS: Correctable Errors collector initialized. [ 3.492501] AVX version of gcm_enc/dec engaged. [ 3.494381] AES CTR mode by8 optimization enabled [ 3.569029] sched_clock: Marking stable (3569012032, 0)->(4535077676, -966065644) [ 3.572526] registered taskstats version 1 [ 3.574662] Loading compiled-in X.509 certificates [ 3.576242] zswap: loaded using pool lzo/zbud [ 3.599843] Key type big_key registered [ 3.611317] Key type encrypted registered [ 3.612759] ima: No TPM chip found, activating TPM-bypass! [ 3.615195] ima: Allocated hash algorithm: sha1 [ 3.617022] ima: No architecture policies found [ 3.618851] evm: Initialising EVM extended attributes: [ 3.622555] evm: security.selinux [ 3.623986] evm: security.ima [ 3.625056] evm: security.capability [ 3.625866] evm: HMAC attrs: 0x1 [ 3.628184] rtc_cmos 00:05: setting system clock to 2026-08-27 14:30:41 UTC (1787841041) [ 3.634342] debug: unmapping init [mem 0xffffffff90a03000-0xffffffff90bfffff] [ 3.637420] debug: unmapping init [mem 0xffffffff8f782000-0xffffffff8fa58fff] [ 3.644179] Write protecting the kernel read-only data: 28672k [ 3.648332] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 3.650993] debug: unmapping init [mem 0xffffffff8e714000-0xffffffff8e7fffff] [ 3.684376] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.692888] systemd[1]: Detected virtualization kvm. [ 3.694897] systemd[1]: Detected architecture x86-64. [ 3.696751] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.722491] systemd[1]: No hostname configured. [ 3.724352] systemd[1]: Set hostname to . [ 3.726587] random: systemd: uninitialized urandom read (16 bytes read) [ 3.729272] systemd[1]: Initializing machine ID from random generator. [ 3.766170] random: ln: uninitialized urandom read (6 bytes read) [ 3.878668] random: systemd: uninitialized urandom read (16 bytes read) [ 3.880914] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.888318] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 3.899098] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.636787] device-mapper: uevent: version 1.0.3 [ 4.639089] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.597331] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 5.712817] random: fast init done [ 5.927367] scsi host0: ata_piix [ 5.946713] scsi host1: ata_piix [ 5.950754] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.953195] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.063527] dracut-initqueue[595]: RTNETLINK answers: File exists [ 11.132213] random: crng init done [ 11.133398] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.471530] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.642600] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.266525] SELinux: Disabled at runtime. [ 15.339313] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.349691] systemd[1]: Detected virtualization kvm. [ 15.351937] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.279909] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.283682] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.293995] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.302633] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.307985] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.316363] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.328213] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. Mounting Huge Pages File System...[ 16.535321] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started Flush Journal to Persistent Storage. [ 17.763788] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 19.109050] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.225532] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.519671] hrtimer: interrupt took 4764946 ns [ 19.830628] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.853498] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 23.887972] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit)[ 24.335995] NFS: Registering the id_resolver key type [ 24.338172] Key type id_resolver registered [ 24.341237] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg113-server login: [ 92.471458] libcfs: loading out-of-tree module taints kernel. [ 92.661442] Key type ._llcrypt registered [ 92.672779] Key type .llcrypt registered [ 92.961897] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 121.539521] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 124.076833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.103284] alg: No test for adler32 (adler32-zlib) [ 126.384092] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 128.273392] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 130.247173] Key type lgssc registered [ 132.311155] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.823230] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 198.138940] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 216.369943] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 216.430046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.794713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 217.867561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 218.015912] Lustre: lustre-MDT0000: new disk, initializing [ 218.125400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.153574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 223.380820] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.566497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.693195] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 239.739620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 239.743527] Lustre: Skipped 1 previous similar message [ 239.834802] Lustre: lustre-MDT0001: new disk, initializing [ 239.901374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 239.940894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 239.953729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 245.415500] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.912496] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 263.614883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.135331] Lustre: lustre-OST0000: new disk, initializing [ 264.139507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 264.150302] Lustre: 8459:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.278909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 271.835387] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 274.005835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 274.011995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 274.145980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 289.196637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 289.426377] Lustre: lustre-OST0001: new disk, initializing [ 289.436275] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 289.447979] Lustre: 9531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 289.548832] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 296.816036] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.997270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 297.017062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 297.120541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 309.495528] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 317.242343] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 323.667644] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing check_logdir /tmp/testlogs/ [ 329.364700] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing yml_node [ 335.126980] Lustre: DEBUG MARKER: Client: 2.17.57.98 [ 337.893929] Lustre: DEBUG MARKER: MDS: 2.17.57.98 [ 340.628862] Lustre: DEBUG MARKER: OSS: 2.17.57.98 [ 342.900536] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:36:19 EDT 2026 [ 361.802370] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 363.524773] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 378.848827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.859294] Lustre: Skipped 1 previous similar message [ 378.861092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 378.871940] Lustre: Skipped 2 previous similar messages [ 383.464229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 383.470536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.488814] Lustre: Skipped 1 previous similar message [ 383.498265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.120881] Lustre: server umount lustre-MDT0000 complete [ 390.276339] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841428 with bad export cookie 3006974367536719152 [ 390.284078] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 390.292766] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 394.207824] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 394.209532] LustreError: 6530:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 394.209715] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 394.209719] Lustre: Skipped 3 previous similar messages [ 394.222476] Lustre: Skipped 1 previous similar message [ 394.245542] LustreError: 6530:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 395.238924] LustreError: 6528:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.270610] LustreError: 6528:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 397.069096] Lustre: server umount lustre-MDT0001 complete [ 403.659549] Lustre: server umount lustre-OST0000 complete [ 410.261860] Lustre: server umount lustre-OST0001 complete [ 418.603055] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 431.834838] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 481.348890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.631857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 481.664404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 481.744533] Lustre: lustre-MDT0000: new disk, initializing [ 481.821600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.835401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 486.417895] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 502.542599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.788187] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 502.967790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 502.985337] Lustre: Skipped 1 previous similar message [ 503.184710] Lustre: lustre-MDT0001: new disk, initializing [ 503.363231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 503.413701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 503.437401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 511.104410] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 526.084241] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 533.476950] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 540.897660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 541.377750] Lustre: lustre-OST0000: new disk, initializing [ 541.384639] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 541.396612] Lustre: 17061:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 541.503698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 542.488040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 542.505476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 542.696072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 547.719840] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 558.294579] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 563.840964] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 564.151287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 569.151342] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 569.324941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 573.414406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 573.421302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.422400] LustreError: Skipped 1 previous similar message [ 573.424891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.424901] Lustre: Skipped 3 previous similar messages [ 573.445438] Lustre: Skipped 1 previous similar message [ 577.796669] Lustre: server umount lustre-OST0000 complete [ 583.651700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.662244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 583.679088] Lustre: Skipped 1 previous similar message [ 583.694502] Lustre: Skipped 2 previous similar messages [ 588.885337] Lustre: server umount lustre-MDT0000 complete [ 592.663776] LustreError: 15887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841630 with bad export cookie 3006974367536720692 [ 592.667214] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 592.671616] LustreError: 15887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 593.116476] Lustre: server umount lustre-MDT0001 complete [ 597.012672] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:40:33 (1787841633) [ 600.080514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 601.992312] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:40:38 (1787841638) [ 609.947022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.478099] LustreError: 19471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.551192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 615.905807] LustreError: 19472:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.364617] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 621.026650] LustreError: 19471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.946253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.762562] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 645.446342] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 653.448891] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 661.202847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 669.770672] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 681.305575] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 690.149992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 690.171258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.195763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 690.216179] Lustre: Skipped 2 previous similar messages [ 695.761513] Lustre: server umount lustre-OST0000 complete [ 706.529410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 706.557833] Lustre: Skipped 5 previous similar messages [ 707.680417] Lustre: server umount lustre-MDT0000 complete [ 711.678394] LustreError: 19471:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.705016] LustreError: 19471:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 712.445285] LustreError: 19451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841750 with bad export cookie 3006974367536721574 [ 712.458552] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 712.465020] LustreError: 19451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 712.808779] Lustre: server umount lustre-MDT0001 complete [ 726.805627] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 730.054228] Key type lgssc unregistered [ 730.383749] LNet: 23010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.392961] LNetError: 23010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 731.433853] LNet: Removed LNI 192.168.201.113@tcp [ 732.580194] Key type .llcrypt unregistered [ 732.587981] Key type ._llcrypt unregistered [ 758.062303] Key type ._llcrypt registered [ 758.067881] Key type .llcrypt registered [ 758.206492] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 759.305755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 759.415570] alg: No test for adler32 (adler32-zlib) [ 760.456377] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 760.685330] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 762.327165] Key type lgssc registered [ 763.409606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 772.047746] loop: module loaded [ 772.113432] loop0: detected capacity change from 0 to 204800000 [ 772.316951] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 772.335220] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 772.788768] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 779.771292] loop0: detected capacity change from 0 to 204800000 [ 779.992963] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 780.014109] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 780.025928] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 787.355326] loop0: detected capacity change from 0 to 204800000 [ 793.835685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 793.850455] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 795.275602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 795.466408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 795.663358] Lustre: lustre-MDT0000: new disk, initializing [ 795.804475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 795.842768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 802.484139] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 814.017960] loop1: detected capacity change from 0 to 204800000 [ 822.655676] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 823.135690] Lustre: lustre-OST07c6: new disk, initializing [ 823.142644] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 823.149754] Lustre: Skipped 1 previous similar message [ 823.164630] Lustre: 26203:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 823.368422] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 831.121747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 831.131658] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 831.164597] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 832.175794] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 838.554874] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 838.561250] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 838.566270] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 838.571918] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 838.576789] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 838.582603] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 857.802574] Lustre: 25188:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 857.812045] Lustre: 25188:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 857.823059] Lustre: 25188:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 857.833395] Lustre: 25188:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 857.853401] Lustre: 25188:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 857.866413] Lustre: 25188:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 863.632957] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 863.646300] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 863.654396] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 863.664272] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 863.669547] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 863.679067] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 863.689468] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 863.700718] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 863.713711] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 863.722787] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 863.733150] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 863.744389] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 867.054647] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 867.069484] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 867.079408] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 867.096415] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 867.106330] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 867.120337] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 867.128886] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 867.136142] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 867.142930] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 867.149160] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 867.156853] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 867.162037] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.033544] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 872.045237] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 872.051860] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 872.061517] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.074328] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 872.080530] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.091757] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 872.098700] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.107159] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 872.118046] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 872.126516] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 872.136764] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 881.155091] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 881.167242] Lustre: 25189:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 881.177043] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 881.189895] Lustre: 25189:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 881.200189] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 881.210390] Lustre: 25189:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 881.222447] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 881.244566] Lustre: 25189:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 881.266783] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 881.285640] Lustre: 25189:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 881.307670] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 881.334509] Lustre: 25189:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 897.332773] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 897.348335] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 897.356528] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 897.363437] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 897.374452] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 897.382392] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 897.388992] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 897.396762] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 897.411484] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 897.421786] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 897.431091] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 897.436058] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 933.865327] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 933.885797] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.935283] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 938.469889] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 939.936613] Lustre: server umount lustre-OST07c6 complete [ 951.621643] Lustre: server umount lustre-MDT0000 complete [ 970.804973] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 973.516789] Key type lgssc unregistered [ 973.817612] LNet: 28773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.828362] LNetError: 28773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.865391] LNet: Removed LNI 192.168.201.113@tcp [ 974.793680] Key type .llcrypt unregistered [ 974.795904] Key type ._llcrypt unregistered [ 994.843890] Key type ._llcrypt registered [ 994.849247] Key type .llcrypt registered [ 994.988750] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 996.194125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 996.378333] alg: No test for adler32 (adler32-zlib) [ 997.563499] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 997.795118] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 999.488502] Key type lgssc registered [ 1000.388790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1008.688240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.725552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1010.441155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1015.700543] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1020.906316] LustreError: 30045:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1020.933288] LustreError: 30045:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1025.607863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.127450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.658517] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1040.564458] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1047.352762] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1055.816270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1056.184227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.442940] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1072.410966] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1080.758636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1101.285468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1101.290435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.294820] LustreError: Skipped 1 previous similar message [ 1101.298044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1101.327630] Lustre: Skipped 1 previous similar message [ 1101.354359] Lustre: Skipped 1 previous similar message [ 1102.869048] Lustre: server umount lustre-OST0000 complete [ 1107.936708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1107.954537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.986722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.522189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.523763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.789819] Lustre: server umount lustre-MDT0000 complete [ 1116.641242] LustreError: 30985:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.674481] LustreError: 30985:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1118.574117] LustreError: 30025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842156 with bad export cookie 4259661227756108632 [ 1118.577537] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.593403] LustreError: 30025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1119.304945] Lustre: server umount lustre-MDT0001 complete [ 1133.761739] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1137.612980] Key type lgssc unregistered [ 1138.052888] LNet: 33807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.062753] LNetError: 33807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.111632] LNet: Removed LNI 192.168.201.113@tcp [ 1140.313628] Key type .llcrypt unregistered [ 1140.320646] Key type ._llcrypt unregistered [ 1160.579398] Key type ._llcrypt registered [ 1160.583799] Key type .llcrypt registered [ 1160.716896] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1161.891994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1162.059341] alg: No test for adler32 (adler32-zlib) [ 1163.160463] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1163.460917] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1165.247288] Key type lgssc registered [ 1166.694619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1176.605727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1176.640791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.153767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1183.397414] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1188.320673] LustreError: 35080:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.349996] LustreError: 35080:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1193.281826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.442405] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1193.673627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1199.900581] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1208.848670] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1216.724128] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1225.235171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.455773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.421853] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1239.344487] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1249.040123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1264.613461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.613984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1264.646501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.652405] LustreError: Skipped 1 previous similar message [ 1267.851710] Lustre: server umount lustre-OST0000 complete [ 1274.849656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.861843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1274.871306] Lustre: Skipped 2 previous similar messages [ 1274.878677] Lustre: Skipped 1 previous similar message [ 1278.410435] Lustre: server umount lustre-MDT0000 complete [ 1279.969886] LustreError: 35079:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.995261] LustreError: 35079:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1282.532877] LustreError: 35059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842320 with bad export cookie 7516846213456781401 [ 1282.535698] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.547925] LustreError: 35059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1283.024984] Lustre: server umount lustre-MDT0001 complete [ 1295.903294] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1298.676057] Key type lgssc unregistered [ 1299.034053] LNet: 38640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.044768] LNetError: 38640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.059078] LNet: Removed LNI 192.168.201.113@tcp [ 1300.105712] Key type .llcrypt unregistered [ 1300.111637] Key type ._llcrypt unregistered [ 1327.811391] Key type ._llcrypt registered [ 1327.814209] Key type .llcrypt registered [ 1327.921968] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1329.411838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1329.443111] alg: No test for adler32 (adler32-zlib) [ 1330.680889] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1331.069638] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1332.834671] Key type lgssc registered [ 1334.400553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1343.964101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1343.973660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.551564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.956215] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1355.746174] LustreError: 40058:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.777234] LustreError: 40058:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1360.871503] LustreError: 40057:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.475978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.940400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.697240] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1376.931472] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1385.132095] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1393.893728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1394.366548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.705280] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1412.313888] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1415.961196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1425.127995] Lustre: server umount lustre-MDT0000 complete [ 1426.402641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.423600] LustreError: 40054:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.434948] Lustre: Skipped 1 previous similar message [ 1426.480756] LustreError: 40054:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1429.218754] LustreError: 40039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842467 with bad export cookie 15972266564781513804 [ 1429.222587] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1429.232173] LustreError: 40039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1429.544447] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [ 1429.554544] LustreError: 40053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1431.522095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.531600] Lustre: Skipped 1 previous similar message [ 1431.823437] Lustre: server umount lustre-MDT0001 complete [ 1444.409346] LustreError: 40038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.13@tcp arrived at 1787842482 with bad export cookie 15972266564781514469 [ 1444.449084] LustreError: 40038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1453.950871] Lustre: server umount lustre-OST0000 complete [ 1473.872908] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1476.969489] Key type lgssc unregistered [ 1477.303766] LNet: 43943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.321845] LNetError: 43943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1477.354732] LNet: Removed LNI 192.168.201.113@tcp [ 1478.451095] Key type .llcrypt unregistered [ 1478.455042] Key type ._llcrypt unregistered [ 1506.327443] Key type ._llcrypt registered [ 1506.329991] Key type .llcrypt registered [ 1506.435091] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1507.761696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1507.839372] alg: No test for adler32 (adler32-zlib) [ 1508.895322] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1509.125438] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1510.754145] Key type lgssc registered [ 1511.810749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1520.145917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1520.159118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.594558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.701790] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1531.874565] LustreError: 45364:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.896661] LustreError: 45364:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1535.743324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.041805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1540.526278] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1548.841405] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1555.666767] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1564.167698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1564.647589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.710894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1571.332353] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1579.564185] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1587.169901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1587.178652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1587.206264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1591.269018] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1591.270746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1591.289145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.293308] Lustre: Skipped 1 previous similar message [ 1592.328407] LustreError: 47187:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1592.492178] Lustre: server umount lustre-OST0000 complete [ 1611.362382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.370833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1611.386401] Lustre: Skipped 1 previous similar message [ 1611.410899] Lustre: Skipped 2 previous similar messages [ 1617.802852] Lustre: server umount lustre-MDT0000 complete [ 1617.893755] LustreError: 45361:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.909823] LustreError: 45361:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1622.031610] LustreError: 45345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842659 with bad export cookie 11055402020680864973 [ 1622.040139] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1622.047538] LustreError: 45345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1636.320073] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1636.613655] Lustre: server umount lustre-MDT0001 complete [ 1650.111698] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1652.604493] Key type lgssc unregistered [ 1652.859840] LNet: 49077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.866310] LNetError: 49077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.881188] LNet: Removed LNI 192.168.201.113@tcp [ 1653.698942] Key type .llcrypt unregistered [ 1653.702484] Key type ._llcrypt unregistered [ 1678.851767] Key type ._llcrypt registered [ 1678.856519] Key type .llcrypt registered [ 1679.009755] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1680.302253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.495193] alg: No test for adler32 (adler32-zlib) [ 1681.628968] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1681.853436] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1683.559171] Key type lgssc registered [ 1684.698536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1692.387666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1692.409614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.126111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1698.843301] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1703.398093] LustreError: 50496:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1703.433745] LustreError: 50496:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1707.937202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.398902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1713.326580] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1721.558491] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1727.945700] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1735.244923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1735.540878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.611135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1741.944689] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1748.840967] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1757.329714] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1761.127898] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1773.582225] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2711h00m09s [ 1775.667557] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1780.194612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1780.204865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.229072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.270114] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1781.276623] Lustre: Skipped 2 previous similar messages [ 1782.378946] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 1782.388277] Lustre: Skipped 3 previous similar messages [ 1783.884049] Lustre: server umount lustre-MDT0000 complete [ 1787.364187] LustreError: 52263:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.384168] LustreError: 52263:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1787.561453] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1788.911108] LustreError: 50492:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.509615] LustreError: 50491:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1793.530512] LustreError: 50491:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1793.802576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1793.917616] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1794.164631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.226107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1795.185086] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1799.485649] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1803.570679] Lustre: DEBUG MARKER: Wait for df (36740) ... [ 1805.467406] Lustre: DEBUG MARKER: done [ 1809.382370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1809.388696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1809.397116] Lustre: Skipped 2 previous similar messages [ 1809.401511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1810.926352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1815.486181] Lustre: server umount lustre-OST0000 complete [ 1821.156074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.167647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.177437] Lustre: Skipped 1 previous similar message [ 1821.188475] Lustre: Skipped 2 previous similar messages [ 1825.438620] Lustre: server umount lustre-MDT0000 complete [ 1828.321740] LustreError: 50495:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.342266] LustreError: 50495:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1829.802599] LustreError: 50475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842867 with bad export cookie 720604166422498167 [ 1829.808323] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1829.818460] LustreError: 50475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1830.514208] Lustre: server umount lustre-MDT0001 complete [ 1843.379832] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 1846.337944] Key type lgssc unregistered [ 1846.714688] LNet: 55436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.723233] LNetError: 55436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.748771] LNet: Removed LNI 192.168.201.113@tcp [ 1847.623553] Key type .llcrypt unregistered [ 1847.630706] Key type ._llcrypt unregistered [ 1888.370564] Key type ._llcrypt registered [ 1888.374921] Key type .llcrypt registered [ 1888.470138] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 1889.812562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1889.822930] alg: No test for adler32 (adler32-zlib) [ 1891.027214] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1891.258088] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 1892.927576] Key type lgssc registered [ 1894.014470] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1902.210368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1902.251466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.589834] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1903.622440] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1903.877592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.570846] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1921.256196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.459665] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1921.499817] Lustre: 57641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1921.718120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1926.300457] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1935.068992] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1942.203529] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1951.405725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1951.746096] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1952.011726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.554642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1959.212131] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 1969.109445] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1976.326880] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2711h03m32s [ 1978.034555] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1986.602623] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1989.481541] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1989.551105] LustreError: 57647:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2009.633933] Lustre: lustre-MDT0000: Client 3be8d8d4-bf54-475c-8488-95d9999f1d61 (at 192.168.201.13@tcp) reconnecting [ 2029.591161] LustreError: 57647:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2031.804365] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2034.473601] Lustre: DEBUG MARKER: done [ 2044.386242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2044.397443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.415685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2044.916116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2044.935303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2044.953606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2046.775449] Lustre: server umount lustre-OST0000 complete [ 2055.139996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.151684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.153804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2055.163962] Lustre: Skipped 1 previous similar message [ 2055.185813] Lustre: Skipped 1 previous similar message [ 2062.305886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2062.317070] Lustre: Skipped 3 previous similar messages [ 2065.375304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2065.660196] Lustre: server umount lustre-MDT0000 complete [ 2070.008697] LustreError: 57631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843107 with bad export cookie 7641363637691021454 [ 2070.012816] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2070.022366] LustreError: 57631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2070.601934] Lustre: server umount lustre-MDT0001 complete [ 2083.754432] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2086.666885] Key type lgssc unregistered [ 2087.104349] LNet: 61977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.140319] LNetError: 61977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.169730] LNet: Removed LNI 192.168.201.113@tcp [ 2089.235194] Key type .llcrypt unregistered [ 2089.236815] Key type ._llcrypt unregistered [ 2137.584472] Key type ._llcrypt registered [ 2137.586619] Key type .llcrypt registered [ 2137.696870] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2138.729499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2138.786678] alg: No test for adler32 (adler32-zlib) [ 2139.944644] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2140.163898] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2141.839428] Key type lgssc registered [ 2143.059411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2146.319228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2146.342833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.638754] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2147.676132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2147.832419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.849299] Lustre: Failing over lustre-MDT0000 [ 2149.270414] Lustre: server umount lustre-MDT0000 complete [ 2156.433969] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:06:32 (1787843192) [ 2164.308415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.737928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.213934] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2177.964353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.131315] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2178.164605] Lustre: 64787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2178.417389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2183.145621] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2192.509725] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2200.128085] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2209.176343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2209.448183] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2209.964856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.083302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2217.563429] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2225.685132] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2232.757808] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2239.969758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2239.984307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.015768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.019991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2242.026613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.396821] Lustre: server umount lustre-MDT0000 complete [ 2248.771894] LustreError: 64778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843286 with bad export cookie 16529676586626183654 [ 2248.780471] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2248.783695] LustreError: 64778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2249.198762] Lustre: server umount lustre-MDT0001 complete [ 2252.516647] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2257.025817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2265.254491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.639971] Lustre: 63712:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843290/real 1787843290] req@00000000b6ca9baa x1874689448257920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2268.675945] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2268.704210] Lustre: Skipped 1 previous similar message [ 2269.664160] Lustre: 63710:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843291/real 1787843291] req@000000007aafc078 x1874689448258048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2273.759304] Lustre: 63710:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843295/real 1787843295] req@00000000111ff35e x1874689448258176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2273.838302] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe56535470df1fac9 [ 2273.856950] Lustre: MGC192.168.201.113@tcp: Connection restored to 0@lo (at 0@lo) [ 2274.176608] LustreError: 66635:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.192411] LustreError: 66635:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2274.289626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.393851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2277.287383] LustreError: 66636:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.311896] LustreError: 68556:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.332116] LustreError: 68556:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2278.784143] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2279.329993] Lustre: 63711:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843301/real 1787843301] req@00000000e94f8dab x1874689448258560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2279.363036] Lustre: 63711:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2283.501889] LustreError: 68558:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.536662] LustreError: 68558:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2284.451285] Lustre: 63709:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843306/real 1787843306] req@0000000020b23dc1 x1874689448258688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2288.618872] LustreError: 68557:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.643651] LustreError: 68557:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2289.113228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.428557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2294.692563] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2294.784530] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2297.906278] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2303.793646] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2311.660439] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2320.356308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2320.365183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.379691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2320.385376] Lustre: Skipped 2 previous similar messages [ 2323.429523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2323.450616] Lustre: Skipped 2 previous similar messages [ 2325.802547] Lustre: server umount lustre-MDT0000 complete [ 2328.603775] LustreError: 68556:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.640081] LustreError: 68556:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2330.863225] LustreError: 64780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843368 with bad export cookie 16529676586626185929 [ 2330.867120] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2330.877582] LustreError: 64780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2331.421495] Lustre: server umount lustre-MDT0001 complete [ 2338.986471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2349.715291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.049081] Lustre: 63711:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843371/real 1787843371] req@00000000c1ee22a2 x1874689448288384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787843387 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2350.097301] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.115445] Lustre: Skipped 2 previous similar messages [ 2355.736372] LustreError: 66636:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.830272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2355.910426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2360.920241] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2369.737109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.153147] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2376.051536] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2385.510383] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.938374] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2400.737138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2400.744459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.762561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.034697] Lustre: server umount lustre-MDT0000 complete [ 2408.931713] LustreError: 66635:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.951972] LustreError: 66635:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 2410.601138] LustreError: 64779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843448 with bad export cookie 16529676586626187308 [ 2410.604680] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2410.614188] LustreError: 64779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2411.260764] Lustre: server umount lustre-MDT0001 complete [ 2419.082685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2424.228061] Lustre: DEBUG MARKER: files compared the same [ 2437.377424] Lustre: server umount lustre-OST0000 complete [ 2456.585912] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2460.104818] Key type lgssc unregistered [ 2460.499097] LNet: 74523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2460.506571] LNetError: 74523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2460.521418] LNet: Removed LNI 192.168.201.113@tcp [ 2461.372334] Key type .llcrypt unregistered [ 2461.375573] Key type ._llcrypt unregistered [ 2486.275784] Key type ._llcrypt registered [ 2486.278080] Key type .llcrypt registered [ 2486.459754] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2487.587729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2487.623210] alg: No test for adler32 (adler32-zlib) [ 2488.904242] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2489.174662] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2490.895800] Key type lgssc registered [ 2492.247300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2500.658736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2500.675661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.261230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2506.895166] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2512.865627] LustreError: 75955:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.891893] LustreError: 75955:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2515.325947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.536324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2519.759797] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2527.989797] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2534.841843] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2542.989676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2543.272105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.391257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2551.058979] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug +malloc all [ 2560.739624] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2571.231985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2571.236531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.244508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2571.261244] Lustre: Skipped 1 previous similar message [ 2571.279850] Lustre: Skipped 1 previous similar message [ 2575.875977] Lustre: server umount lustre-OST0000 complete [ 2581.472519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.483848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.593218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.596878] Lustre: Skipped 2 previous similar messages [ 2591.712522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2595.296944] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2595.467974] Lustre: server umount lustre-MDT0000 complete [ 2596.835893] LustreError: 76682:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.845526] LustreError: 76682:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2600.909327] LustreError: 75935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843638 with bad export cookie 15634257716451418101 [ 2600.912643] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.930676] LustreError: 75935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2601.569588] Lustre: server umount lustre-MDT0001 complete [ 2614.445811] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2617.350948] Key type lgssc unregistered [ 2617.619388] LNet: 79493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2617.624894] LNetError: 79493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2617.643966] LNet: Removed LNI 192.168.201.113@tcp [ 2618.766803] Key type .llcrypt unregistered [ 2618.768967] Key type ._llcrypt unregistered [ 2642.609251] Key type ._llcrypt registered [ 2642.611626] Key type .llcrypt registered [ 2642.700535] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2644.011029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2644.038769] alg: No test for adler32 (adler32-zlib) [ 2645.122744] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2645.336824] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2647.007637] Key type lgssc registered [ 2648.244651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2656.513878] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2656.551036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2672.159579] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2677.279532] LustreError: 80885:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [ 2703.967716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2710.117647] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2717.572394] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2725.492233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2725.663846] Lustre: *** cfs_fail_loc=706, val=0*** [ 2726.013182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.083983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2730.937936] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2731.491681] LustreError: 80911:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.529692] LustreError: 80911:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2736.612183] LustreError: 80910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.660825] LustreError: 81971:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.687305] LustreError: 81971:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2739.722647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.994809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2744.596657] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2752.298431] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2758.236983] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2760.676372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2760.680167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.701261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.708978] Lustre: Skipped 1 previous similar message [ 2765.808297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2765.826814] Lustre: Skipped 1 previous similar message [ 2766.926970] Lustre: server umount lustre-OST0000 complete [ 2776.038148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2776.040270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.042756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.062159] Lustre: Skipped 2 previous similar messages [ 2777.439033] Lustre: server umount lustre-MDT0000 complete [ 2780.873488] LustreError: 80892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843818 with bad export cookie 16729245678268818970 [ 2780.884554] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2780.890044] LustreError: 80892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2781.204746] Lustre: server umount lustre-MDT0001 complete [ 2791.662793] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 2793.717678] Key type lgssc unregistered [ 2793.947595] LNet: 84504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2793.956442] LNetError: 84504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2794.985399] LNet: Removed LNI 192.168.201.113@tcp [ 2795.772818] Key type .llcrypt unregistered [ 2795.777302] Key type ._llcrypt unregistered [ 2825.815195] Key type ._llcrypt registered [ 2825.819132] Key type .llcrypt registered [ 2825.928663] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 2827.025974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2827.140297] alg: No test for adler32 (adler32-zlib) [ 2828.259434] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2828.561623] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 2830.271549] Key type lgssc registered [ 2832.027646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2840.594975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2840.613749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.317940] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2853.689750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.364817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2858.562757] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2859.495251] LustreError: 86608:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2859.534214] LustreError: 86608:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2864.609595] LustreError: 86612:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.636125] LustreError: 86612:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2869.025584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2869.486349] LustreError: 86608:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.512422] LustreError: 86608:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2869.533170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2874.868515] LustreError: 86611:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.894192] LustreError: 86611:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2875.727160] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2879.975254] LustreError: 86608:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2880.016284] LustreError: 86608:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2885.018861] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2885.054077] Lustre: MGS: Client 784f4c86-3c7a-4d30-9794-88d7d4ffd5c1 (at 0@lo) reconnecting [ 2885.068909] LustreError: 86622:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2885.367910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.495868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2890.172821] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 2906.111882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.113847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2906.127087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2906.149076] Lustre: Skipped 1 previous similar message [ 2911.200819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2911.219288] Lustre: Skipped 1 previous similar message [ 2911.519520] Lustre: server umount lustre-OST0000 complete [ 2921.441928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2921.447583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.460659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2921.469758] Lustre: Skipped 2 previous similar messages [ 2921.476090] Lustre: Skipped 1 previous similar message [ 2923.153590] Lustre: server umount lustre-MDT0000 complete [ 2926.560855] LustreError: 88527:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.585149] LustreError: 88527:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2927.950543] Lustre: server umount lustre-MDT0001 complete [ 2931.781517] Lustre: server umount MGS complete [ 2944.916991] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:19:41 (1787843981) [ 2976.629184] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 2989.505382] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3031.933883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.572938] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3047.248602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3047.788412] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3047.913645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3048.120828] Lustre: lustre-MDT0001: new disk, initializing [ 3048.269207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3048.326867] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3048.334751] Lustre: 93801:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3048.350743] LustreError: 93801:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3054.078323] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3066.064037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3066.321781] Lustre: lustre-OST0000: new disk, initializing [ 3066.328044] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3066.343747] Lustre: 94565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3066.460103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3073.927682] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3074.145525] Lustre: 94866:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004d0ee3d6 x1874690169675648/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787844128 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3084.178336] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3092.688153] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.708884] Lustre: MGS: Client fa6f9460-a29f-4fc1-8218-1449f6551f14 (at 0@lo) reconnecting [ 3092.787462] Lustre: 95612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3092.802994] LustreError: 93787:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3092.835785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3092.925379] Lustre: lustre-MDT0000: new disk, initializing [ 3093.104958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.147594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3093.179579] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3094.243180] Lustre: 95636:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000875323de x1874690169682944/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787844148 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3098.190062] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3099.362436] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3103.413646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3103.422551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3103.454672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3129.312517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3129.315753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.339899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.923649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.931735] Lustre: Skipped 2 previous similar messages [ 3143.135899] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3143.381330] Lustre: server umount lustre-OST0000 complete [ 3149.280393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3149.283919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.292965] LustreError: Skipped 1 previous similar message [ 3149.294994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3149.295031] Lustre: Skipped 2 previous similar messages [ 3149.317933] Lustre: Skipped 2 previous similar messages [ 3153.666402] Lustre: server umount lustre-MDT0000 complete [ 3154.404907] LustreError: 96053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.436913] LustreError: 96053:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3164.774946] Lustre: server umount lustre-MDT0001 complete [ 3169.065295] Lustre: server umount MGS complete [ 3181.866902] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:23:38 (1787844218) [ 3202.515535] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3205.549379] Key type lgssc unregistered [ 3206.011932] LNet: 98144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3206.020029] LNetError: 98144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3206.035530] LNet: Removed LNI 192.168.201.113@tcp [ 3207.065136] Key type .llcrypt unregistered [ 3207.070745] Key type ._llcrypt unregistered [ 3227.934470] Key type ._llcrypt registered [ 3227.936398] Key type .llcrypt registered [ 3228.041666] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3229.547589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3229.576938] alg: No test for adler32 (adler32-zlib) [ 3230.720487] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 3231.010073] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3232.664385] Key type lgssc registered [ 3234.414618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3243.686631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3243.719911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.294474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3249.967856] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3255.786662] LustreError: 99366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.814722] LustreError: 99366:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3260.188094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.508950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3265.777713] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3275.120711] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3282.671373] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3290.223258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3290.552433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3291.569833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3296.930725] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3305.444356] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3316.192638] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3316.199171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.231270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3321.323831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3321.339609] Lustre: Skipped 1 previous similar message [ 3322.263097] Lustre: server umount lustre-OST0000 complete [ 3326.946064] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3326.954993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3326.989728] Lustre: Skipped 1 previous similar message [ 3326.998645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.011269] Lustre: Skipped 1 previous similar message [ 3331.554117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3331.557431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3331.590175] Lustre: Skipped 1 previous similar message [ 3332.904411] Lustre: server umount lustre-MDT0000 complete [ 3336.694575] LustreError: 99366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.727340] LustreError: 99366:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3337.060758] LustreError: 99345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844374 with bad export cookie 7677844569846352159 [ 3337.076497] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3337.087770] LustreError: 99345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3337.400719] Lustre: server umount lustre-MDT0001 complete [ 3352.824328] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3356.828438] Key type lgssc unregistered [ 3357.191189] LNet: 102911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3357.206942] LNetError: 102911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3357.250363] LNet: Removed LNI 192.168.201.113@tcp [ 3358.220127] Key type .llcrypt unregistered [ 3358.222292] Key type ._llcrypt unregistered [ 3376.421623] Key type ._llcrypt registered [ 3376.424461] Key type .llcrypt registered [ 3376.595891] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3378.103286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3378.225592] alg: No test for adler32 (adler32-zlib) [ 3379.516604] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 3379.812156] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3381.543179] Key type lgssc registered [ 3383.351127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3391.167732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3391.180574] LustreError: 103989:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3391.213485] LustreError: 103996:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3391.223244] LustreError: 103989:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4973 [ 3391.245188] LustreError: 103989:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3391.261194] LustreError: 103989:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3391.280883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.923520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3402.214876] LustreError: 104035:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.223467] LustreError: 104035:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3402.771197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3403.056440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3408.429608] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3413.822135] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing lsmod [ 3416.584488] LustreError: 105202:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3416.604873] LustreError: 105203:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3416.610808] LustreError: 105202:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 3416.618075] LustreError: 105202:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3416.629978] LustreError: 105202:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3416.641537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3417.061275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3418.094606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3423.212412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.215876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3423.228339] Lustre: Skipped 2 previous similar messages [ 3423.241498] Lustre: Skipped 2 previous similar messages [ 3426.717430] Lustre: server umount lustre-MDT0000 complete [ 3427.379503] LustreError: 104035:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.395953] LustreError: 104035:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3430.732035] LustreError: 104010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844468 with bad export cookie 3740188673898805069 [ 3430.753263] LustreError: 104010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3430.760785] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3431.583447] LustreError: 104465:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008b6fe71a x1874690747421568/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3431.791332] Lustre: server umount lustre-MDT0001 complete [ 3445.487247] Lustre: server umount lustre-OST0000 complete [ 3456.195756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3456.712945] LustreError: 106267:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3456.861406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.124565] LustreError: 106268:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3462.830801] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3466.212561] LustreError: 106267:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3473.818118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.283166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3481.614475] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3491.554689] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3497.790454] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3505.684687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3506.152606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.972901] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3521.167023] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3527.667548] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:97) [ 3539.938456] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3539.945844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3539.964707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3543.012535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3543.015447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3544.864147] Lustre: server umount lustre-OST0000 complete [ 3551.203343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3551.218714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.253543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3551.269032] Lustre: Skipped 1 previous similar message [ 3555.896916] Lustre: server umount lustre-MDT0000 complete [ 3558.371646] LustreError: 106268:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3558.383664] LustreError: 106268:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3560.388728] LustreError: 106247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844598 with bad export cookie 3740188673898806000 [ 3560.389927] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3560.404281] LustreError: 106247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3560.836633] Lustre: server umount lustre-MDT0001 complete [ 3574.965703] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3578.313183] Key type lgssc unregistered [ 3578.601594] LNet: 109828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3578.611961] LNetError: 109828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3578.630505] LNet: Removed LNI 192.168.201.113@tcp [ 3579.634848] Key type .llcrypt unregistered [ 3579.637457] Key type ._llcrypt unregistered [ 3608.942130] Key type ._llcrypt registered [ 3608.945438] Key type .llcrypt registered [ 3609.171396] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3610.758266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3610.827921] alg: No test for adler32 (adler32-zlib) [ 3611.914819] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 3612.412374] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3614.295788] Key type lgssc registered [ 3615.633169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3625.975959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3626.006748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.623383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.377846] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3637.739080] LustreError: 111256:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.783947] LustreError: 111256:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3642.853442] LustreError: 111255:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3645.443332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3645.745450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3650.093382] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3659.660743] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3665.707766] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3673.673752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3674.018894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.176315] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3689.465092] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3697.593605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3705.828096] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3710.432473] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3716.066617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3716.069447] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.082942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.121935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3718.123962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.145091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.205382] Lustre: Skipped 1 previous similar message [ 3723.239807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3723.249464] Lustre: Skipped 1 previous similar message [ 3728.355238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3728.367416] Lustre: Skipped 1 previous similar message [ 3729.887169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3730.226836] Lustre: server umount lustre-OST0000 complete [ 3738.084635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3738.092409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.118813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3748.836995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3748.848096] Lustre: Skipped 4 previous similar messages [ 3749.343482] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3749.620957] Lustre: server umount lustre-MDT0000 complete [ 3753.959774] LustreError: 112271:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3753.985351] LustreError: 112271:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3754.346936] LustreError: 111238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844792 with bad export cookie 1438961616816980596 [ 3754.349956] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3754.364652] LustreError: 111238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3754.746680] Lustre: server umount lustre-MDT0001 complete [ 3769.283717] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3773.457746] Key type lgssc unregistered [ 3773.861675] LNet: 114925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3773.885843] LNetError: 114925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3773.898808] LNet: Removed LNI 192.168.201.113@tcp [ 3775.296337] Key type .llcrypt unregistered [ 3775.298731] Key type ._llcrypt unregistered [ 3797.044842] Key type ._llcrypt registered [ 3797.047529] Key type .llcrypt registered [ 3797.154752] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3798.863573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3798.909039] alg: No test for adler32 (adler32-zlib) [ 3799.935884] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 3800.270123] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3802.031288] Key type lgssc registered [ 3803.818909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3813.088842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3813.154565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3814.844061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.935954] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3825.128431] LustreError: 116148:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3825.149366] LustreError: 116148:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3830.256469] LustreError: 116147:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3830.339885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3830.910856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3837.373199] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3849.959826] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3857.396540] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3865.795527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3866.232210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3874.457472] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 3883.580697] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3884.757413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3897.848944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3897.854228] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.869101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3899.168987] Lustre: server umount lustre-OST0000 complete [ 3905.504607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.508922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3905.519228] Lustre: Skipped 1 previous similar message [ 3905.541945] Lustre: Skipped 1 previous similar message [ 3910.626813] LustreError: 116148:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.888800] Lustre: server umount lustre-MDT0000 complete [ 3915.178428] LustreError: 116129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844953 with bad export cookie 250141439071351443 [ 3915.181046] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3915.189889] LustreError: 116129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3925.821873] Lustre: server umount lustre-MDT0001 complete [ 3939.721940] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 3943.681665] Key type lgssc unregistered [ 3944.191038] LNet: 119705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3944.206918] LNetError: 119705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3944.228402] LNet: Removed LNI 192.168.201.113@tcp [ 3945.627594] Key type .llcrypt unregistered [ 3945.630841] Key type ._llcrypt unregistered [ 3976.375670] Key type ._llcrypt registered [ 3976.378222] Key type .llcrypt registered [ 3976.582430] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 3978.332573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3978.349734] alg: No test for adler32 (adler32-zlib) [ 3979.421149] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 3979.660764] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 3981.343720] Key type lgssc registered [ 3982.617978] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3991.630474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3991.672797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3993.227715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3999.370162] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4003.809835] LustreError: 121128:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4003.829181] LustreError: 121128:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4008.933050] LustreError: 121127:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4010.701585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.397085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4018.480824] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4029.764931] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4037.068803] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4045.792053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4046.246670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4052.533525] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4061.982145] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4064.688111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:163 to 0x240000400:193) [ 4073.731541] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4079.417712] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4079.429411] Lustre: Skipped 2 previous similar messages [ 4082.345319] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4082.408157] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4085.885557] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4085.892303] Lustre: Skipped 2 previous similar messages [ 4091.514930] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4091.517739] Lustre: Skipped 1 previous similar message [ 4098.713676] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.13@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4098.722526] Lustre: Skipped 2 previous similar messages [ 4100.279157] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.13@tcp 192.168.0.[2,10]@tcp [ 4112.532647] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.13/24@tcp in log lustre-MDT0000 [ 4112.538121] Lustre: Skipped 2 previous similar messages [ 4121.199249] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.13/24@tcp [ 4121.210494] Lustre: Skipped 1 previous similar message [ 4128.224429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4128.235396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4128.260104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4129.253257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4129.258329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.206696] Lustre: server umount lustre-OST0000 complete [ 4139.496222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4139.503194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.514079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4139.526388] Lustre: Skipped 1 previous similar message [ 4139.540256] Lustre: Skipped 1 previous similar message [ 4144.609921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.616338] Lustre: Skipped 2 previous similar messages [ 4149.735080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4149.746651] Lustre: Skipped 1 previous similar message [ 4153.311130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4153.676966] Lustre: server umount lustre-MDT0000 complete [ 4154.850746] LustreError: 121127:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4154.869901] LustreError: 121127:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4158.252697] LustreError: 121108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845196 with bad export cookie 17605955245957749499 [ 4158.268803] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4158.270084] LustreError: 121108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4158.640061] Lustre: server umount lustre-MDT0001 complete [ 4172.928688] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4176.212097] Key type lgssc unregistered [ 4176.480649] LNet: 126079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4176.487629] LNetError: 126079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4176.500312] LNet: Removed LNI 192.168.201.113@tcp [ 4177.576683] Key type .llcrypt unregistered [ 4177.583856] Key type ._llcrypt unregistered [ 4212.340719] Key type ._llcrypt registered [ 4212.342610] Key type .llcrypt registered [ 4212.474318] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4214.167839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4214.216650] alg: No test for adler32 (adler32-zlib) [ 4215.355841] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 4215.508187] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4217.151204] Key type lgssc registered [ 4218.573770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4227.929051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4227.947452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4229.645509] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4229.659359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4229.678100] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.13/24@tcp [ 4234.808776] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4239.844956] LustreError: 127643:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4239.859434] LustreError: 127643:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4244.965963] LustreError: 127644:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4246.165880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4246.810670] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4246.866083] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.13/24@tcp [ 4246.908858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4253.390455] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4264.499696] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4272.728040] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4280.816219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4281.184291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.031262] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4296.702486] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4299.655617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:201 to 0x240000400:225) [ 4312.043644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4312.050044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4312.065624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4315.106896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4315.112191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.116871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4315.149412] Lustre: Skipped 1 previous similar message [ 4317.604990] Lustre: server umount lustre-OST0000 complete [ 4323.812345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4323.816219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.826141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.497401] Lustre: server umount lustre-MDT0000 complete [ 4330.472995] LustreError: 127643:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4330.499966] LustreError: 127643:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4332.725344] LustreError: 127626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845370 with bad export cookie 5584264542367308820 [ 4332.729660] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4332.738841] LustreError: 127626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4332.970498] Lustre: server umount lustre-MDT0001 complete [ 4346.347413] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4350.458232] Key type lgssc unregistered [ 4350.914589] LNet: 131239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4350.937092] LNetError: 131239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4351.974875] LNet: Removed LNI 192.168.201.113@tcp [ 4353.520335] Key type .llcrypt unregistered [ 4353.529659] Key type ._llcrypt unregistered [ 4385.485925] Key type ._llcrypt registered [ 4385.491820] Key type .llcrypt registered [ 4385.699302] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 4403.661596] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4405.875739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4405.952949] alg: No test for adler32 (adler32-zlib) [ 4407.364694] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 4407.865895] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4409.799234] Key type lgssc registered [ 4412.045120] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4459.132198] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4459.195907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.591943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4460.648238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4460.768608] Lustre: lustre-MDT0000: new disk, initializing [ 4460.856389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4460.876444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4466.769772] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4480.193513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.455143] Lustre: 134642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4480.575164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4480.583262] Lustre: Skipped 1 previous similar message [ 4480.757423] Lustre: lustre-MDT0001: new disk, initializing [ 4480.930645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4480.995238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4481.012958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4486.831451] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4498.606312] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4505.641825] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4514.178942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4514.465839] Lustre: lustre-OST0000: new disk, initializing [ 4514.489806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4514.498161] Lustre: 136613:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4514.646555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4521.602292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4521.623703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4521.824109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4522.197979] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4534.682639] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4540.614878] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4540.890721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4546.256463] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4546.544973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4552.675925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4552.675995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.687683] LustreError: Skipped 1 previous similar message [ 4552.689147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4552.737992] Lustre: Skipped 1 previous similar message [ 4555.383554] Lustre: server umount lustre-OST0000 complete [ 4562.912277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4562.914027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4562.928272] Lustre: Skipped 1 previous similar message [ 4562.943393] Lustre: Skipped 2 previous similar messages [ 4566.423049] Lustre: server umount lustre-MDT0000 complete [ 4570.716863] LustreError: 134634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845608 with bad export cookie 8052716345415672742 [ 4570.719252] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4570.734931] LustreError: 134634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4571.502650] Lustre: server umount lustre-MDT0001 complete [ 4581.924514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4582.861695] LustreError: 138832:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4582.885301] LustreError: 138832:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4582.972352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4588.000717] LustreError: 138833:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.949160] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4593.123922] LustreError: 138832:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4597.990987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4598.376769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4603.169935] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4611.886450] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4617.893753] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4625.509646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4625.892450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4632.561930] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4641.819430] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4648.177488] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4648.495082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4653.235582] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4653.577303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4663.847319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4664.056911] Lustre: lustre-OST0001: new disk, initializing [ 4664.061813] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4664.068970] Lustre: 142208:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4664.148286] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4672.820152] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4673.807707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4673.834367] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4674.029231] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4684.456253] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4684.711955] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4691.180470] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4691.622879] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4699.123118] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a7c51b23800.ost_server_uuid 50 [ 4701.300834] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a7c51b23800.ost_server_uuid in IDLE state after 0 sec [ 4714.983462] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4714.999501] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4715.034426] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4718.077228] Lustre: server umount lustre-OST0001 complete [ 4720.098052] LustreError: 141847:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4720.128332] LustreError: 141847:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4725.232349] LustreError: 140655:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4725.232747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.233919] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4725.233926] LustreError: Skipped 1 previous similar message [ 4725.245733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4725.245742] Lustre: Skipped 1 previous similar message [ 4725.272319] LustreError: 140655:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4725.325757] Lustre: Skipped 2 previous similar messages [ 4729.878274] Lustre: server umount lustre-OST0000 complete [ 4735.456661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.475166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4735.488323] Lustre: Skipped 2 previous similar messages [ 4741.752172] Lustre: server umount lustre-MDT0000 complete [ 4745.705386] LustreError: 138833:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.736750] LustreError: 138833:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4746.533249] LustreError: 138814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845784 with bad export cookie 8052716345415673624 [ 4746.534142] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4746.550904] LustreError: 138814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4747.250191] Lustre: server umount lustre-MDT0001 complete [ 4772.989197] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 4775.941503] Key type lgssc unregistered [ 4776.271652] LNet: 145217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4776.279108] LNetError: 145217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4776.298308] LNet: Removed LNI 192.168.201.113@tcp [ 4777.070527] Key type .llcrypt unregistered [ 4777.072471] Key type ._llcrypt unregistered [ 4822.252646] Key type ._llcrypt registered [ 4822.254682] Key type .llcrypt registered [ 4822.370099] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 4841.272642] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 4842.710747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4843.063361] alg: No test for adler32 (adler32-zlib) [ 4844.262559] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 4844.517368] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 4846.215488] Key type lgssc registered [ 4847.565397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4887.872942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4887.913314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4889.274669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4889.329618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4889.395353] Lustre: lustre-MDT0000: new disk, initializing [ 4889.464137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4889.476280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4895.259713] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4907.698103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.053137] Lustre: 149376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4908.202130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4908.210729] Lustre: Skipped 1 previous similar message [ 4908.325901] Lustre: lustre-MDT0001: new disk, initializing [ 4908.467255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4908.523331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4908.536313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4913.788943] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4925.223179] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4931.357390] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4938.531249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4938.895548] Lustre: lustre-OST0000: new disk, initializing [ 4938.899697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4938.905190] Lustre: 151342:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4938.990135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4945.546625] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4945.977870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4945.987371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4946.143305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4956.225887] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4962.262287] Lustre: Failing over lustre-OST0000 [ 4962.546925] Lustre: server umount lustre-OST0000 complete [ 4980.136080] Lustre: 147596:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846001/real 1787846001] req@00000000fc4520b5 x1874692283600128/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787846017 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4980.187412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4982.239685] Lustre: 147596:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846004/real 1787846004] req@0000000079fee9fa x1874692283600640/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787846020 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4982.752284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4982.793948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4983.240710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4983.258863] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4988.319177] Lustre: 147598:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846009/real 1787846009] req@0000000059a48df9 x1874692283602432/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787846025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4988.360472] Lustre: 147598:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4990.374682] Lustre: 147596:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846012/real 1787846012] req@0000000065091ef7 x1874692283603328/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787846028 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4990.389894] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4991.081855] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 4992.539642] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4992.540424] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4995.547217] Lustre: Failing over lustre-MDT0000 [ 4995.573343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4995.587440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.610253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4996.014735] Lustre: server umount lustre-MDT0000 complete [ 4996.575352] Lustre: 147597:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846018/real 1787846018] req@000000004ab097d4 x1874692283604352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787846034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4996.621160] Lustre: 147597:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5000.679228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5000.694221] Lustre: Skipped 1 previous similar message [ 5001.195993] LustreError: 149382:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.226460] LustreError: 149382:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 5005.795412] LustreError: 149381:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5005.811633] LustreError: 149381:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5010.913538] LustreError: 149387:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5010.955963] LustreError: 149387:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 5016.035703] LustreError: 151725:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5016.067683] LustreError: 151725:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 5016.991156] Lustre: 147597:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846038/real 1787846038] req@0000000023be8b58 x1874692283617408/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787846054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5017.009393] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5017.101828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5022.192399] LustreError: 151725:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5022.226176] LustreError: 151725:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 5026.341129] LustreError: 147595:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000096f9f9ce x1874692283622784/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5026.777100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5026.843166] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5027.818280] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5031.723796] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5031.954353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5031.966148] Lustre: Skipped 1 previous similar message [ 5031.997705] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5032.046716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5042.155181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5042.169811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5042.181918] Lustre: Skipped 1 previous similar message [ 5047.266970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5047.281901] Lustre: Skipped 2 previous similar messages [ 5048.434390] Lustre: server umount lustre-OST0000 complete [ 5057.509482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.513058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5057.519936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5057.540115] Lustre: Skipped 1 previous similar message [ 5059.984762] Lustre: server umount lustre-MDT0000 complete [ 5062.648234] LustreError: 151725:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5062.668353] LustreError: 151725:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 5064.274462] LustreError: 149367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846102 with bad export cookie 13626292052030767805 [ 5064.286464] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5064.291333] LustreError: 149367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5064.600460] Lustre: server umount lustre-MDT0001 complete [ 5076.011575] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 5078.691634] Key type lgssc unregistered [ 5078.963634] LNet: 155002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5078.979425] LNetError: 155002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5079.000925] LNet: Removed LNI 192.168.201.113@tcp [ 5079.894732] Key type .llcrypt unregistered [ 5079.897746] Key type ._llcrypt unregistered [ 5121.907738] Key type ._llcrypt registered [ 5121.909448] Key type .llcrypt registered [ 5122.046766] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5123.166062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5123.270080] alg: No test for adler32 (adler32-zlib) [ 5124.426519] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 5124.632510] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 5126.319589] Key type lgssc registered [ 5127.729170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5135.828932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5135.882393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5137.529801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5142.649358] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5148.130044] LustreError: 156526:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5148.153378] LustreError: 156526:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5153.255686] LustreError: 156527:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5154.321744] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5158.370567] LustreError: 156526:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5158.495312] LustreError: 156555:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5158.503898] LustreError: 156555:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5158.513113] LustreError: 156555:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 5159.142505] Lustre: server umount lustre-MDT0000 complete [ 5176.019290] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5188.145737] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5230.992247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5231.175883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5231.197380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5231.263742] Lustre: lustre-MDT0000: new disk, initializing [ 5231.336029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5231.359621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5236.377422] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5248.831455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.014673] Lustre: 159863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5249.105050] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5249.110648] Lustre: Skipped 1 previous similar message [ 5249.176453] Lustre: lustre-MDT0001: new disk, initializing [ 5249.269667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5249.329478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5249.347877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5254.026938] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5265.751217] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5272.701971] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5282.295554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5282.641519] Lustre: lustre-OST0000: new disk, initializing [ 5282.646254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5282.653838] Lustre: 161831:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5282.779688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5290.047815] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5292.108175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5292.117806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5292.250227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5301.594362] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5306.226665] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5308.326397] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:59:04 (1787846344) [ 5587.424877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5587.442851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.462916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5589.473910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.477926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5589.496992] Lustre: Skipped 1 previous similar message [ 5593.949321] Lustre: server umount lustre-MDT0000 complete [ 5594.592282] LustreError: 162456:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.601294] LustreError: 162456:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5598.341389] LustreError: 159853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846636 with bad export cookie 3461738639294393684 [ 5598.344183] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5598.349800] LustreError: 159853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5599.713246] LustreError: 160937:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5599.714921] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5599.756507] Lustre: Skipped 1 previous similar message [ 5599.761379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5603.746957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5604.726875] Lustre: server umount lustre-MDT0001 complete [ 5623.263480] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5623.516107] Lustre: server umount lustre-OST0000 complete [ 5636.151364] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5652.842504] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5702.555632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5702.812813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5702.848166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5702.976562] Lustre: lustre-MDT0000: new disk, initializing [ 5703.067158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.089388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5709.162647] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5722.883756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5723.185416] Lustre: 166535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5723.257086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5723.261713] Lustre: Skipped 1 previous similar message [ 5723.397155] Lustre: lustre-MDT0001: new disk, initializing [ 5723.573925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5723.650211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5723.658257] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5730.478606] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5742.368704] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5749.790956] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5757.818958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5758.254389] Lustre: lustre-OST0000: new disk, initializing [ 5758.265595] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5758.274308] Lustre: 168503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5758.384298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5766.494102] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5767.808967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5767.833559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5767.922916] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5778.670084] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5784.840320] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5785.165512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5792.217792] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5792.657374] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5798.882841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5798.886190] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5798.895850] LustreError: Skipped 1 previous similar message [ 5798.902058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5798.934078] Lustre: Skipped 1 previous similar message [ 5807.038346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5807.057525] Lustre: Skipped 4 previous similar messages [ 5809.121779] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5809.335639] Lustre: server umount lustre-OST0000 complete [ 5815.779282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5815.789510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.807934] Lustre: server umount lustre-MDT0000 complete [ 5822.440682] LustreError: 166540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.460242] LustreError: 166540:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5824.631959] LustreError: 166525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846862 with bad export cookie 3461738639294552514 [ 5824.639065] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5824.640882] LustreError: 166525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5824.962562] Lustre: server umount lustre-MDT0001 complete [ 5836.389729] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:07:52 (1787846872) [ 5855.787517] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 5867.290637] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 5907.845719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5907.993917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5908.025746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5908.121933] Lustre: lustre-MDT0000: new disk, initializing [ 5908.184878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5908.196880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5913.939166] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5925.922108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.056818] Lustre: 173267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5926.427785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5931.230260] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5942.890730] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5949.818311] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5957.636538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5958.122398] Lustre: lustre-OST0000: new disk, initializing [ 5958.125230] Lustre: Skipped 1 previous similar message [ 5958.129094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5958.132585] Lustre: Skipped 2 previous similar messages [ 5958.136979] Lustre: 175247:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5965.380969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5965.400088] Lustre: Skipped 1 previous similar message [ 5965.413932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5965.594982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5965.922503] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 5976.141190] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5986.275058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5986.279053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5986.284237] LustreError: Skipped 1 previous similar message [ 5986.285824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5986.296691] Lustre: Skipped 2 previous similar messages [ 5986.304712] Lustre: Skipped 3 previous similar messages [ 5990.418588] Lustre: server umount lustre-OST0000 complete [ 6001.204253] Lustre: server umount lustre-MDT0000 complete [ 6001.632440] LustreError: 173272:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.668945] LustreError: 173272:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6005.320458] LustreError: 173257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847043 with bad export cookie 3461738639294553396 [ 6005.331601] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6005.341632] LustreError: 173257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6005.798946] Lustre: server umount lustre-MDT0001 complete [ 6017.933922] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:10:54 (1787847054) [ 6040.530041] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 6052.001373] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6097.816821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6098.096203] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6098.149502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6098.304075] Lustre: lustre-MDT0000: new disk, initializing [ 6098.414261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6098.417775] Lustre: Skipped 2 previous similar messages [ 6098.439908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6104.714597] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6116.834185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.985511] Lustre: 179568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6117.157621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6121.782617] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6132.201411] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6138.469988] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6145.086117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6145.529988] Lustre: 181547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6152.838114] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6154.893155] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6155.041492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6163.361255] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6175.714038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6175.720981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6175.727825] Lustre: Skipped 2 previous similar messages [ 6175.740595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6175.753919] Lustre: Skipped 3 previous similar messages [ 6177.128848] Lustre: server umount lustre-OST0000 complete [ 6195.167595] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6195.439745] Lustre: server umount lustre-MDT0000 complete [ 6199.489539] LustreError: 179563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847237 with bad export cookie 3461738639294554481 [ 6199.504033] LustreError: 179563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6199.504660] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6199.828894] Lustre: server umount lustre-MDT0001 complete [ 6211.884953] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 6214.791275] Key type lgssc unregistered [ 6215.034680] LNet: 183407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6215.048087] LNetError: 183407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6215.083476] LNet: Removed LNI 192.168.201.113@tcp [ 6216.239056] Key type .llcrypt unregistered [ 6216.241814] Key type ._llcrypt unregistered [ 6246.378418] Key type ._llcrypt registered [ 6246.380958] Key type .llcrypt registered [ 6246.533622] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6247.854880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6248.102196] alg: No test for adler32 (adler32-zlib) [ 6249.399031] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 6249.773464] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 6251.594859] Key type lgssc registered [ 6253.113947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6261.884369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6261.903605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.542276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6269.140345] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6274.027410] LustreError: 184826:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6274.058461] LustreError: 184826:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6278.781637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.159934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6284.453953] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6292.977303] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6299.091767] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6307.381690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6308.049798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6309.120535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6314.648685] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6321.790721] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6344.168648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6344.176056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.190126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6344.207339] Lustre: Skipped 1 previous similar message [ 6346.602883] Lustre: server umount lustre-OST0000 complete [ 6350.816966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6350.823722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6350.836386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6350.841477] Lustre: Skipped 1 previous similar message [ 6354.404944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6354.406308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6354.434762] Lustre: Skipped 1 previous similar message [ 6356.295902] Lustre: server umount lustre-MDT0000 complete [ 6359.528220] LustreError: 184826:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6359.554320] LustreError: 184826:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6360.447104] LustreError: 184809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847398 with bad export cookie 18133438932316063799 [ 6360.461636] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6360.835334] Lustre: server umount lustre-MDT0001 complete [ 6373.191164] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 6376.260886] Key type lgssc unregistered [ 6376.606806] LNet: 188386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6376.620325] LNetError: 188386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6376.630303] LNet: Removed LNI 192.168.201.113@tcp [ 6377.729494] Key type .llcrypt unregistered [ 6377.733886] Key type ._llcrypt unregistered [ 6402.442579] Key type ._llcrypt registered [ 6402.446537] Key type .llcrypt registered [ 6402.545863] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6403.526283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6403.573899] alg: No test for adler32 (adler32-zlib) [ 6404.718911] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 6404.978839] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 6406.631201] Key type lgssc registered [ 6407.633138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6414.315064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6414.331168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6415.752898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6420.208378] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6426.083226] LustreError: 189803:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6426.101895] LustreError: 189803:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6428.295275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6428.648097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6433.595319] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6441.610771] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6447.969956] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6456.006906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6456.540457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6460.672424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6462.463829] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6470.742914] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6481.384504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6481.392781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6481.411044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6484.008753] Lustre: lustre-OST0000: Not available for connect from 192.168.201.13@tcp (stopping) [ 6484.454265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6484.472104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6487.190219] Lustre: server umount lustre-OST0000 complete [ 6494.572556] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a7c5191b800.ost_server_uuid 50 [ 6496.206489] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6515.188582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6515.200263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6515.202203] Lustre: Skipped 1 previous similar message [ 6515.230815] Lustre: Skipped 3 previous similar messages [ 6516.979227] Lustre: server umount lustre-MDT0000 complete [ 6520.288683] LustreError: 190792:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6520.307143] LustreError: 190792:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6520.627483] LustreError: 189784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847558 with bad export cookie 2633437086811441067 [ 6520.634213] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6520.643474] LustreError: 189784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6520.939654] Lustre: server umount lustre-MDT0001 complete [ 6529.681812] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:19:26 (1787847566) [ 6535.287900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.625465] LustreError: 193718:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.733773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6539.638329] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6540.769162] LustreError: 193719:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6545.889165] LustreError: 193718:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.448091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.830759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6552.369670] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6560.277883] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6566.086909] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6572.947198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6573.287383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6579.736093] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6580.470292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6587.663904] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6594.399195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6594.639971] Lustre: lustre-OST0001: new disk, initializing [ 6594.644361] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6594.648828] Lustre: 196561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6594.728847] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6601.244572] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6603.839762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6603.845103] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6603.921138] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6610.159258] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6613.473125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6613.477740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6613.496946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6619.118872] LustreError: 196456:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6619.138423] LustreError: 196456:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6619.261620] Lustre: server umount lustre-OST0000 complete [ 6625.901360] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6626.073566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6630.381924] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6630.574616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6636.524709] LustreError: 195540:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6636.547208] LustreError: 195540:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 6644.704908] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6644.709845] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6644.712856] Lustre: Skipped 2 previous similar messages [ 6644.719813] Lustre: Skipped 3 previous similar messages [ 6650.345871] Lustre: server umount lustre-OST0001 complete [ 6654.944984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6654.953667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6654.955300] Lustre: Skipped 1 previous similar message [ 6654.961783] Lustre: Skipped 3 previous similar messages [ 6669.281835] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6669.553461] Lustre: server umount lustre-MDT0000 complete [ 6670.306175] LustreError: 193714:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6670.318636] LustreError: 193714:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 6673.780972] LustreError: 193698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847711 with bad export cookie 2633437086811442131 [ 6673.786628] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6673.788995] LustreError: 193698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6674.239553] Lustre: server umount lustre-MDT0001 complete [ 6697.780180] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:22:14 (1787847734) [ 6704.940727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.238628] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6705.290727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6705.486085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6710.784160] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6719.416158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6719.555740] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6719.620311] Lustre: 200000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6724.464829] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6732.904707] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6739.850821] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6748.343707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6748.533611] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6754.472165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6754.769342] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6763.921859] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6771.336468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6771.396564] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6771.631576] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6771.640771] Lustre: Skipped 2 previous similar messages [ 6779.668765] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6780.206880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6789.183774] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6795.237245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6795.245802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6795.268094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6795.279130] Lustre: Skipped 4 previous similar messages [ 6798.958447] Lustre: server umount lustre-OST0000 complete [ 6800.371227] LustreError: 201847:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.224442] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6821.879390] Lustre: server umount lustre-OST0001 complete [ 6831.585563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6831.591904] Lustre: Skipped 9 previous similar messages [ 6832.387126] Lustre: server umount lustre-MDT0000 complete [ 6836.824968] LustreError: 199992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847874 with bad export cookie 2633437086811443552 [ 6836.829867] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6836.845166] LustreError: 199992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6837.487588] Lustre: server umount lustre-MDT0001 complete [ 6864.519447] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:25:00 (1787847900) [ 6884.042518] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 6896.617535] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 6943.980659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6944.273549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6944.304639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6944.382192] Lustre: lustre-MDT0000: new disk, initializing [ 6944.484982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6944.513293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6950.050454] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6962.495622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.644824] Lustre: 208097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6962.709496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6962.714938] Lustre: Skipped 1 previous similar message [ 6962.799766] Lustre: lustre-MDT0001: new disk, initializing [ 6962.983847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6963.005749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6968.645955] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 6980.345840] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6987.737608] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6996.226115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6996.526621] Lustre: lustre-OST0000: new disk, initializing [ 6996.529947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6996.541086] Lustre: 210068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7003.498353] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7004.758468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7004.772216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7004.904054] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7015.563576] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7021.758725] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7022.064290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7028.695450] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7029.251054] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7035.874126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7035.880402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7035.901249] Lustre: Skipped 6 previous similar messages [ 7035.921724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7035.936699] LustreError: Skipped 1 previous similar message [ 7046.111167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7046.478573] Lustre: server umount lustre-OST0000 complete [ 7057.548278] Lustre: server umount lustre-MDT0000 complete [ 7061.473434] LustreError: 208111:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7061.503821] LustreError: 208111:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [ 7062.234107] LustreError: 208090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848100 with bad export cookie 2633437086811445337 [ 7062.243390] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7062.250459] LustreError: 208090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7062.632269] Lustre: server umount lustre-MDT0001 complete [ 7072.800593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7078.746379] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7090.074753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7096.138777] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7106.537453] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7113.715743] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7122.004647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7129.924520] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7139.846401] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7145.708406] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7146.058242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7151.326311] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7151.636423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7154.144179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7154.155275] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7154.173193] Lustre: Skipped 3 previous similar messages [ 7164.392524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7164.401261] Lustre: Skipped 14 previous similar messages [ 7167.967223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7168.180251] Lustre: server umount lustre-OST0000 complete [ 7176.072635] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7181.456182] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7212.625307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7213.067611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7213.075610] Lustre: Skipped 5 previous similar messages [ 7219.189630] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7219.221926] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7219.233936] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7220.741781] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7230.080724] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7232.180631] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7239.653888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7239.669488] LustreError: Skipped 1 previous similar message [ 7243.138394] Lustre: server umount lustre-OST0000 complete [ 7254.230525] Lustre: server umount lustre-MDT0000 complete [ 7259.050155] LustreError: 212269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848296 with bad export cookie 2633437086811446212 [ 7259.058906] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7259.072116] LustreError: 212269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7259.552364] Lustre: server umount lustre-MDT0001 complete [ 7274.053271] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:31:50 (1787848310) [ 7282.336807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7289.995586] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7301.875932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7308.690094] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7318.331619] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7326.477356] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7334.884233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7344.979672] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7354.692482] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7360.355172] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7360.618313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7365.836970] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7366.045651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7375.883674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7376.182287] Lustre: lustre-OST0001: new disk, initializing [ 7376.187170] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7376.195317] Lustre: 221637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7384.204074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7384.213254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7384.378495] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7384.689915] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7396.877813] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7402.352954] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7402.571372] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7408.684755] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7409.214188] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7415.277539] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7415.281722] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7415.291284] LustreError: Skipped 1 previous similar message [ 7415.336863] Lustre: Skipped 5 previous similar messages [ 7417.535964] Lustre: server umount lustre-OST0001 complete [ 7420.401947] LustreError: 220084:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7420.433767] LustreError: 220084:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 7425.892826] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7426.175433] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7431.661220] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7431.989335] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7462.359784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7468.020518] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7468.039993] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7468.044206] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7468.064520] Lustre: Skipped 1 previous similar message [ 7469.574234] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7478.331918] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7480.356237] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7483.386401] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7483.404844] Lustre: Skipped 7 previous similar messages [ 7488.856717] Lustre: server umount lustre-OST0001 complete [ 7522.271184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7522.693107] Lustre: server umount lustre-OST0000 complete [ 7535.111540] Lustre: server umount lustre-MDT0000 complete [ 7539.774397] LustreError: 218242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848577 with bad export cookie 2633437086811447185 [ 7539.778591] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7539.796164] LustreError: 218242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7546.366703] Lustre: server umount lustre-MDT0001 complete [ 7575.699542] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:36:51 (1787848611) [ 7583.662079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.920307] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7583.945957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7588.981459] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7598.732135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7598.795618] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7598.817937] Lustre: 226885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7604.396337] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7613.428806] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7621.096364] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7628.990037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7629.260750] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7636.510215] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7645.928544] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7657.038801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.131402] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7666.271322] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7676.434920] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7678.873510] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7685.658172] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7686.008722] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7692.251703] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7692.421464] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7699.972876] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a7c45a6a000.ost_server_uuid 50 [ 7702.031683] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a7c45a6a000.ost_server_uuid in IDLE state after 0 sec [ 7703.587572] Lustre: Permanently deactivating lustre-OST0001 [ 7703.599624] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7707.684245] Lustre: Permanently reactivating lustre-OST0001 [ 7707.701531] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7707.705447] Lustre: Skipped 2 previous similar messages [ 7712.740992] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7712.752623] LustreError: Skipped 3 previous similar messages [ 7712.757282] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7712.775619] Lustre: Skipped 6 previous similar messages [ 7718.613731] Lustre: server umount lustre-OST0001 complete [ 7728.547429] Lustre: server umount lustre-OST0000 complete [ 7738.913953] Lustre: server umount lustre-MDT0000 complete [ 7743.187863] LustreError: 226880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848781 with bad export cookie 2633437086811448466 [ 7743.192383] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7743.196966] LustreError: 226880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7743.686922] Lustre: server umount lustre-MDT0001 complete [ 7770.998143] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:40:07 (1787848807) [ 7779.074340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7779.450419] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7779.490318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7779.505667] Lustre: Skipped 2 previous similar messages [ 7779.682243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.692905] Lustre: Skipped 9 previous similar messages [ 7784.750238] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7795.642425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7795.937801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7796.038296] Lustre: 233582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7802.798422] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7812.848650] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7819.943631] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7828.947714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.230267] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7829.245532] Lustre: Permanently deactivating lustre-OST0000 [ 7829.262782] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7837.612881] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7838.754984] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7847.272916] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7856.210269] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7856.284561] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7864.178586] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 7865.713594] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7874.591656] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7876.592204] Lustre: Permanently reactivating lustre-OST0000 [ 7876.612255] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7876.618521] Lustre: Skipped 2 previous similar messages [ 7885.425467] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7885.444479] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 47 seconds. I think it's dead, and I am evicting it. exp 000000006530d97b, cur 1787848923 deadline 1787848906 last 1787848876 [ 7885.533629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7902.193524] Lustre: server umount lustre-OST0001 complete [ 7912.603276] Lustre: server umount lustre-OST0000 complete [ 7923.602602] Lustre: server umount lustre-MDT0000 complete [ 7928.186985] LustreError: 236465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848966 with bad export cookie 2633437086811450139 [ 7928.192833] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7928.198690] LustreError: 236465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7928.895669] Lustre: server umount lustre-MDT0001 complete [ 7942.818347] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 7945.881659] Key type lgssc unregistered [ 7946.153770] LNet: 238469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7946.164603] LNetError: 238469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7946.190560] LNet: Removed LNI 192.168.201.113@tcp [ 7947.209795] Key type .llcrypt unregistered [ 7947.217327] Key type ._llcrypt unregistered [ 7973.890368] Key type ._llcrypt registered [ 7973.892485] Key type .llcrypt registered [ 7973.980393] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 7975.464017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7975.569996] alg: No test for adler32 (adler32-zlib) [ 7976.814298] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 7977.142654] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 7978.879230] Key type lgssc registered [ 7980.463580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7990.920987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7990.946166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7992.654312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7998.180983] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8003.040599] LustreError: 239942:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8003.066214] LustreError: 239942:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8007.493038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8007.688550] Lustre: Found index 1 for lustre-MDT0001, updating log [ 8007.710169] Lustre: 239930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8007.718768] Lustre: Permanently deactivating lustre-MDT0001 [ 8007.732745] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8008.017602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8010.738400] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8010.751643] LustreError: 239972:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 8010.768466] LustreError: 239972:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 8010.785617] LustreError: 239972:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 8014.007181] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8023.405508] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8029.900599] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8037.228570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8037.661047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8038.726410] LustreError: 241783:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8038.758167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 8044.012439] LustreError: 241782:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.419150] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8049.145446] LustreError: 241782:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8052.275674] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 8055.179890] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8057.358449] LustreError: 242378:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8057.382907] LustreError: 242378:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8064.618294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8065.019521] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8070.144234] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 8070.147853] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 8072.124762] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8081.612953] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8085.153926] Lustre: Permanently reactivating lustre-MDT0001 [ 8085.188560] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 8085.198360] Lustre: Skipped 2 previous similar messages [ 8088.284432] Lustre: lustre-MDT0001: Client 2eb3cf90-4ce3-43bc-97f4-7aa1a6272da2 (at 192.168.201.13@tcp) reconnecting [ 8094.560644] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8096.100929] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8103.162580] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8103.337338] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8108.252149] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8108.597323] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8111.522709] Lustre: Permanently deactivating lustre-MDT0001 [ 8111.543453] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8111.547298] Lustre: Skipped 2 previous similar messages [ 8120.988172] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8128.314352] Lustre: server umount lustre-MDT0000 complete [ 8131.044864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8131.057603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.079670] LustreError: 240686:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8131.105515] LustreError: 240686:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8131.562705] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.597593] Lustre: Skipped 2 previous similar messages [ 8131.981153] LustreError: 243492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849169 with bad export cookie 17977552233983619039 [ 8131.991077] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8131.995163] LustreError: 243492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8138.377926] Lustre: server umount lustre-MDT0001 complete [ 8142.253535] Lustre: server umount lustre-OST0000 complete [ 8154.829321] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:46:31 (1787849191) [ 8166.095932] Lustre: server umount lustre-OST0001 complete [ 8172.654619] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 8183.830899] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8225.601613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8226.024762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8226.035273] Lustre: Skipped 2 previous similar messages [ 8226.096509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8226.264663] Lustre: lustre-MDT0000: new disk, initializing [ 8226.350437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8226.393084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8231.381760] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8241.944139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8242.100676] Lustre: 248224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8242.156906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8242.164070] Lustre: Skipped 1 previous similar message [ 8242.267194] Lustre: lustre-MDT0001: new disk, initializing [ 8242.366399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8242.415081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8242.436490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8247.231101] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8258.077784] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8264.385487] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8272.215237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8272.578647] Lustre: lustre-OST0000: new disk, initializing [ 8272.586713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8272.600979] Lustre: 250190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8272.700037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8279.217328] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8280.163980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8280.172535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8280.312818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8291.012627] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8293.782758] LustreError: 248229:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8303.855671] LustreError: 248229:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8303.914952] LustreError: 250962:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8306.420189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8306.739282] Lustre: lustre-OST0001: new disk, initializing [ 8306.751892] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8306.775315] Lustre: 251308:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8306.996337] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8313.967519] LustreError: 250962:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8314.022409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8314.036502] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8314.246216] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8315.155411] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8326.578741] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8328.692190] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8334.816110] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8334.837658] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.840062] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8334.862445] Lustre: Skipped 1 previous similar message [ 8337.873553] Lustre: server umount lustre-OST0001 complete [ 8339.949475] LustreError: 250172:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8339.985540] LustreError: 250172:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 8357.350735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8357.358979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.361027] LustreError: Skipped 2 previous similar messages [ 8357.367694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8357.367702] Lustre: Skipped 1 previous similar message [ 8357.406853] Lustre: Skipped 1 previous similar message [ 8360.443461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8360.455670] Lustre: Skipped 2 previous similar messages [ 8362.391510] Lustre: server umount lustre-OST0000 complete [ 8370.657336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.665586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.668384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8370.678763] Lustre: Skipped 1 previous similar message [ 8370.713358] Lustre: Skipped 1 previous similar message [ 8373.084184] Lustre: server umount lustre-MDT0000 complete [ 8375.781115] LustreError: 252282:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.800657] LustreError: 252282:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 8377.184208] LustreError: 248214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849415 with bad export cookie 17977552233983621853 [ 8377.199353] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8377.203251] LustreError: 248214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8383.807755] Lustre: server umount lustre-MDT0001 complete [ 8397.166259] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8400.209238] Key type lgssc unregistered [ 8400.443697] LNet: 253350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8400.450485] LNetError: 253350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8400.466744] LNet: Removed LNI 192.168.201.113@tcp [ 8401.281131] Key type .llcrypt unregistered [ 8401.283663] Key type ._llcrypt unregistered [ 8448.822971] Key type ._llcrypt registered [ 8448.824576] Key type .llcrypt registered [ 8448.929640] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8450.822058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8450.842358] alg: No test for adler32 (adler32-zlib) [ 8451.841345] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 8452.042504] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8453.727165] Key type lgssc registered [ 8454.762886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8467.090293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8467.111406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.563966] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8468.590571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8468.767193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8476.330771] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8491.359655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8491.421786] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8491.443422] Lustre: 255561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8491.674849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8502.003488] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8513.633113] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8521.195345] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8529.319288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8529.581484] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8529.813279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8537.659779] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8537.800242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8546.473400] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8604.640812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8604.651296] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.657247] LustreError: Skipped 1 previous similar message [ 8604.666560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8606.510907] Lustre: server umount lustre-OST0000 complete [ 8611.726323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8625.933604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.379594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8633.830521] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8643.459470] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8645.746905] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8645.766213] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8645.769321] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8645.792638] Lustre: Skipped 1 previous similar message [ 8687.074848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8687.099506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8687.107045] Lustre: Skipped 2 previous similar messages [ 8687.119955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8687.164863] Lustre: Skipped 2 previous similar messages [ 8692.280161] Lustre: server umount lustre-OST0000 complete [ 8696.800341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8696.806818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8696.832865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8702.284977] Lustre: server umount lustre-MDT0000 complete [ 8702.446591] LustreError: 255573:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8702.483206] LustreError: 255573:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 8706.245966] LustreError: 255554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849744 with bad export cookie 16975066702468324970 [ 8706.254078] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8706.259600] LustreError: 255554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8706.716426] Lustre: server umount lustre-MDT0001 complete [ 8718.936045] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8721.886590] Key type lgssc unregistered [ 8722.266620] LNet: 260992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8722.276129] LNetError: 260992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8723.300857] LNet: Removed LNI 192.168.201.113@tcp [ 8724.334767] Key type .llcrypt unregistered [ 8724.338903] Key type ._llcrypt unregistered [ 8751.510595] Key type ._llcrypt registered [ 8751.512773] Key type .llcrypt registered [ 8751.618641] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8753.253441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8753.287496] alg: No test for adler32 (adler32-zlib) [ 8754.376419] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 8754.616113] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8756.324722] Key type lgssc registered [ 8757.676468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8766.554533] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8766.579350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8768.192242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.208353] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8778.723725] LustreError: 262409:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8778.740253] LustreError: 262409:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8782.373256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8782.753217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8786.989127] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8794.536944] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8800.520770] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8807.668245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8807.891417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.239182] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8815.102204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8825.894887] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8861.153235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8861.167864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8861.202456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8863.209204] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8863.217807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8863.223695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8867.185890] Lustre: server umount lustre-OST0000 complete [ 8873.458041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8873.465446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8873.489262] Lustre: Skipped 1 previous similar message [ 8873.508018] Lustre: Skipped 2 previous similar messages [ 8877.631263] Lustre: server umount lustre-MDT0000 complete [ 8878.561033] LustreError: 262403:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8878.598316] LustreError: 262403:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8882.493717] LustreError: 262388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849920 with bad export cookie 6201451587880901278 [ 8882.512937] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8883.032805] Lustre: server umount lustre-MDT0001 complete [ 8895.961050] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 8899.239632] Key type lgssc unregistered [ 8899.542731] LNet: 266717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8899.558564] LNetError: 266717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8899.579102] LNet: Removed LNI 192.168.201.113@tcp [ 8900.617868] Key type .llcrypt unregistered [ 8900.621465] Key type ._llcrypt unregistered [ 8920.302259] Key type ._llcrypt registered [ 8920.304717] Key type .llcrypt registered [ 8920.393690] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 8921.767029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8921.790653] alg: No test for adler32 (adler32-zlib) [ 8923.043706] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 8923.386759] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 8925.071559] Key type lgssc registered [ 8926.376307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8934.620104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8934.642324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.202276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.561562] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8946.658458] LustreError: 267938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8946.676977] LustreError: 267938:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8951.537879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8951.828940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8957.367820] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8966.311962] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8972.787398] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8980.133579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8980.430236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8981.502084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8986.811040] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 8993.862742] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9012.206649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9012.212387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9012.214976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9012.224558] Lustre: Skipped 1 previous similar message [ 9012.230053] LustreError: Skipped 1 previous similar message [ 9015.986635] Lustre: server umount lustre-OST0000 complete [ 9022.440452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9022.443810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9022.455961] Lustre: Skipped 1 previous similar message [ 9022.478585] Lustre: Skipped 2 previous similar messages [ 9027.474081] Lustre: server umount lustre-MDT0000 complete [ 9027.552597] LustreError: 267938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9027.576544] LustreError: 267938:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9031.774020] LustreError: 267921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850069 with bad export cookie 6168498347768517932 [ 9031.781639] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9031.796961] LustreError: 267921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9032.248541] Lustre: server umount lustre-MDT0001 complete [ 9045.528556] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9049.173619] Key type lgssc unregistered [ 9049.533269] LNet: 271632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9049.553143] LNetError: 271632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9049.603294] LNet: Removed LNI 192.168.201.113@tcp [ 9050.734933] Key type .llcrypt unregistered [ 9050.744406] Key type ._llcrypt unregistered [ 9079.560524] Key type ._llcrypt registered [ 9079.562720] Key type .llcrypt registered [ 9079.659800] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9081.430135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9081.469245] alg: No test for adler32 (adler32-zlib) [ 9082.590544] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 9082.874184] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9084.575182] Key type lgssc registered [ 9085.796252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9094.564635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9094.590375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9096.193865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9100.745255] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9106.411114] LustreError: 273049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9106.449665] LustreError: 273049:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9109.804947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9110.253870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9114.386781] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9122.882957] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9129.794652] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9138.563437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9138.898134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9141.939053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9145.890832] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9154.377867] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9186.272769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9186.283036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9186.286722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9186.310719] Lustre: Skipped 1 previous similar message [ 9189.512841] Lustre: server umount lustre-OST0000 complete [ 9196.513101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9196.517593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9196.527082] Lustre: Skipped 1 previous similar message [ 9196.530307] Lustre: Skipped 1 previous similar message [ 9200.321581] Lustre: server umount lustre-MDT0000 complete [ 9201.639066] LustreError: 274102:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9201.652541] LustreError: 274102:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9204.669498] LustreError: 273032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850242 with bad export cookie 9330925772135025772 [ 9204.679730] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9204.681924] LustreError: 273032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9205.202777] Lustre: server umount lustre-MDT0001 complete [ 9221.366817] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9224.804685] Key type lgssc unregistered [ 9225.223120] LNet: 277361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9225.233710] LNetError: 277361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9225.262226] LNet: Removed LNI 192.168.201.113@tcp [ 9226.260738] Key type .llcrypt unregistered [ 9226.266631] Key type ._llcrypt unregistered [ 9249.657897] Key type ._llcrypt registered [ 9249.660629] Key type .llcrypt registered [ 9249.775412] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9251.223547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9251.437334] alg: No test for adler32 (adler32-zlib) [ 9252.700436] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 9252.992618] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9254.695183] Key type lgssc registered [ 9255.834766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9263.526828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9263.549869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.150956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9269.747867] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9274.340332] LustreError: 278601:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9274.364718] LustreError: 278601:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9279.034574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9279.318202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9284.745284] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9293.807968] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9301.093626] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9309.107595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9309.642243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9318.367747] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9327.367381] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9334.277355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9354.726820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.739204] Lustre: Skipped 1 previous similar message [ 9354.746177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9354.757440] Lustre: Skipped 1 previous similar message [ 9357.466541] Lustre: server umount lustre-OST0000 complete [ 9364.964198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.982552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9364.986484] Lustre: Skipped 1 previous similar message [ 9368.069184] Lustre: server umount lustre-MDT0000 complete [ 9370.085780] LustreError: 278604:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.108028] LustreError: 278604:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9372.591338] LustreError: 278567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850410 with bad export cookie 209761165252503542 [ 9372.600060] LustreError: 278567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9372.604437] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9373.247713] Lustre: server umount lustre-MDT0001 complete [ 9385.512483] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9388.330892] Key type lgssc unregistered [ 9388.656214] LNet: 282330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9388.668638] LNetError: 282330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9388.688932] LNet: Removed LNI 192.168.201.113@tcp [ 9389.648715] Key type .llcrypt unregistered [ 9389.653503] Key type ._llcrypt unregistered [ 9402.325474] Key type ._llcrypt registered [ 9402.329283] Key type .llcrypt registered [ 9402.437969] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9421.568194] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 9438.649648] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9440.026559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9440.099740] alg: No test for adler32 (adler32-zlib) [ 9441.231820] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 9441.501883] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9443.191184] Key type lgssc registered [ 9444.311649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9480.855052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9480.902722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9482.291821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9482.363531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9482.502777] Lustre: lustre-MDT0000: new disk, initializing [ 9482.587986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9482.613503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9487.636355] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9499.660561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9499.821907] Lustre: 285881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9499.894622] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9499.899187] Lustre: Skipped 1 previous similar message [ 9500.065718] Lustre: lustre-MDT0001: new disk, initializing [ 9500.180640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9500.280293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9500.293702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9505.427419] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9518.159656] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9524.033710] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9533.786210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9534.110776] Lustre: lustre-OST0000: new disk, initializing [ 9534.114566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9534.122398] Lustre: 287853:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9534.195312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9540.728462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9540.747405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9540.869313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9541.638827] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9552.359625] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9558.626759] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9558.998375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9564.201397] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9564.385493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9566.693120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9566.693878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9566.694503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9566.710030] LustreError: Skipped 1 previous similar message [ 9566.743158] Lustre: Skipped 1 previous similar message [ 9571.816913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9571.836183] Lustre: Skipped 1 previous similar message [ 9572.558376] Lustre: server umount lustre-OST0000 complete [ 9582.053477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9582.062489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9582.070116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9582.071057] Lustre: Skipped 1 previous similar message [ 9582.104456] Lustre: Skipped 2 previous similar messages [ 9583.624122] Lustre: server umount lustre-MDT0000 complete [ 9587.704062] LustreError: 285874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850625 with bad export cookie 8004652372686630904 [ 9587.708944] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9587.714481] LustreError: 285874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9588.085388] Lustre: server umount lustre-MDT0001 complete [ 9599.089322] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:10:35 (1787850635) [ 9607.217839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.801082] LustreError: 290266:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9607.838053] LustreError: 290266:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9607.955252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9613.235855] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9613.289106] LustreError: 290267:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9617.389191] LustreError: 290266:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9622.498066] LustreError: 290267:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9623.989480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9624.564351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9630.911195] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9639.257832] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9646.540516] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9654.716181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9655.147467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9661.995945] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9670.740631] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9694.186430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9694.196354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9694.209154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9694.236423] Lustre: Skipped 1 previous similar message [ 9697.328496] Lustre: server umount lustre-OST0000 complete [ 9704.417335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9704.442541] Lustre: Skipped 1 previous similar message [ 9704.466910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9704.482828] Lustre: Skipped 1 previous similar message [ 9708.821330] Lustre: server umount lustre-MDT0000 complete [ 9709.536692] LustreError: 290266:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9709.558658] LustreError: 290266:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9712.693245] LustreError: 290247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850750 with bad export cookie 8004652372686631786 [ 9712.708399] LustreError: 290247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9712.708966] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9713.285467] Lustre: server umount lustre-MDT0001 complete [ 9728.228495] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [ 9731.372250] Key type lgssc unregistered [ 9731.689514] LNet: 293830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9731.698068] LNetError: 293830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9731.739684] LNet: Removed LNI 192.168.201.113@tcp [ 9732.956088] Key type .llcrypt unregistered [ 9732.962426] Key type ._llcrypt unregistered [ 9769.740836] Key type ._llcrypt registered [ 9769.744889] Key type .llcrypt registered [ 9769.905041] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [ 9786.706934] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [ 9787.653723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9787.777720] alg: No test for adler32 (adler32-zlib) [ 9789.063277] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 9789.392928] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [ 9791.111288] Key type lgssc registered [ 9793.888608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9859.803671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9859.853608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9861.292233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9861.335597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9861.463522] Lustre: lustre-MDT0000: new disk, initializing [ 9861.589137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.632444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9866.907297] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9879.431053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.702499] Lustre: 297667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9879.844659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9879.851664] Lustre: Skipped 1 previous similar message [ 9879.973844] Lustre: lustre-MDT0001: new disk, initializing [ 9880.116887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9880.168748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9880.193560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9886.053291] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9896.410753] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9903.061782] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9910.765742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9911.109660] Lustre: lustre-OST2710: new disk, initializing [ 9911.127786] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9911.153896] Lustre: 299634:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9911.346187] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9917.514079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9917.525652] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9917.694712] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9918.454748] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9928.483732] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9936.236335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9936.431478] Lustre: lustre-OST03e8: new disk, initializing [ 9936.435508] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9936.441831] Lustre: 300700:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9936.545652] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9943.084742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9943.094937] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9943.228494] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9943.538345] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [ 9951.908327] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9960.928818] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9961.238521] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9968.464141] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9968.782953] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9974.907609] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9975.260432] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9980.249603] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9980.577138] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9987.553446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9987.568952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9987.584104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9989.601776] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9989.602959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9989.617139] Lustre: Skipped 2 previous similar messages [ 9989.643299] Lustre: Skipped 3 previous similar messages [ 9992.632303] Lustre: server umount lustre-MDT0000 complete [ 9997.110823] LustreError: 300699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851034 with bad export cookie 15248091801208059303 [ 9997.112538] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9997.134092] LustreError: 300699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9997.407313] Lustre: server umount lustre-MDT0001 complete [10011.562393] Lustre: server umount lustre-OST2710 complete [10015.711659] Lustre: 295459:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851037/real 1787851037] req@00000000dee1d616 x1874697468846976/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787851053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10015.758448] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10017.730492] Lustre: server umount lustre-OST03e8 complete [10041.222622] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [10052.960386] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10092.654375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10092.864735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10092.904645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10092.980509] Lustre: lustre-MDT0000: new disk, initializing [10093.059770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10093.080263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10100.090793] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10116.194629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10116.439759] Lustre: 305954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10116.533633] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10116.546568] Lustre: Skipped 1 previous similar message [10116.727956] Lustre: lustre-MDT0001: new disk, initializing [10116.983707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10117.056061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10117.067991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10122.922148] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10133.899434] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10141.391248] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10149.001821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10149.360666] Lustre: lustre-OST0000: new disk, initializing [10149.372177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10149.382170] Lustre: 307928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10149.516279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10157.612973] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10159.288518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10159.297380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10159.464253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10167.882885] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10173.233413] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10173.525813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10179.281552] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10179.533885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10185.186124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10185.201382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10185.217154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10185.220022] Lustre: Skipped 1 previous similar message [10189.317161] Lustre: server umount lustre-OST0000 complete [10195.427576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10195.443664] Lustre: Skipped 1 previous similar message [10195.448561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10200.544869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10200.560746] Lustre: Skipped 2 previous similar messages [10209.247368] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10209.518751] Lustre: server umount lustre-MDT0000 complete [10210.810101] LustreError: 305962:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10210.834123] LustreError: 305962:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [10214.670645] LustreError: 309469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851252 with bad export cookie 15248091801208060997 [10214.683102] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10214.690500] LustreError: 309469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10215.019632] Lustre: server umount lustre-MDT0001 complete [10225.092471] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:21:01 (1787851261) [10226.733565] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10229.062880] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:21:05 (1787851265) [10233.386184] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10267.085367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10267.436684] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10267.468714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10267.674919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10274.294889] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10289.755963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10289.997686] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10290.097932] Lustre: 311787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10297.028213] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10320.941601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10321.220268] Lustre: Denying initial registration attempt from nid 192.168.201.113@tcp, specified as failover [10321.231311] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10321.245104] LustreError: 313246:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10321.253956] LustreError: 313246:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10321.260155] LustreError: 313246:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10321.339759] Lustre: server umount lustre-OST0000 complete [10321.342834] LustreError: 313246:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10329.568610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10329.576662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10329.587323] Lustre: Skipped 2 previous similar messages [10329.605125] Lustre: Skipped 3 previous similar messages [10334.764878] Lustre: server umount lustre-MDT0000 complete [10339.803645] LustreError: 311781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851377 with bad export cookie 15248091801208061886 [10339.806374] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10339.811894] LustreError: 311781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10340.435633] Lustre: server umount lustre-MDT0001 complete [10353.511552] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10356.652146] Key type lgssc unregistered [10357.060762] LNet: 314371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10357.070129] LNetError: 314371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10357.092408] LNet: Removed LNI 192.168.201.113@tcp [10358.159964] Key type .llcrypt unregistered [10358.162362] Key type ._llcrypt unregistered [10372.547737] Key type ._llcrypt registered [10372.552661] Key type .llcrypt registered [10372.685988] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10374.078147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10374.292736] alg: No test for adler32 (adler32-zlib) [10375.555948] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [10376.028241] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10377.695187] Key type lgssc registered [10379.295536] Lustre: Echo OBD driver; http://www.lustre.org/ [10432.786581] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10442.815981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10442.850171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10444.197389] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10444.242575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10444.517632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10450.533457] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10461.108093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10461.327235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10461.391801] Lustre: 317139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10461.812168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10467.932284] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10478.927485] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10486.985913] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10509.900040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10510.464682] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10510.757587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10518.347407] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10528.235670] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10535.910437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10535.916672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10535.925466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10535.941614] Lustre: Skipped 1 previous similar message [10535.964814] Lustre: Skipped 1 previous similar message [10537.694420] Lustre: server umount lustre-OST0000 complete [10543.588266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10543.602135] LustreError: Skipped 1 previous similar message [10543.608117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10543.626433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10546.144769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10546.150207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10551.267066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10551.284404] Lustre: Skipped 2 previous similar messages [10555.363758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10555.378841] Lustre: Skipped 1 previous similar message [10556.383859] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10557.100464] Lustre: server umount lustre-MDT0000 complete [10562.162464] LustreError: 317130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851600 with bad export cookie 2666253386668220724 [10562.167041] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10562.171221] LustreError: 317130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10562.568987] Lustre: server umount lustre-MDT0001 complete [10576.756844] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10580.535509] Key type lgssc unregistered [10580.960068] LNet: 320720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10580.979729] LNetError: 320720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10581.013149] LNet: Removed LNI 192.168.201.113@tcp [10582.625800] Key type .llcrypt unregistered [10582.630085] Key type ._llcrypt unregistered [10614.127842] Key type ._llcrypt registered [10614.131758] Key type .llcrypt registered [10614.275081] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10616.262816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10616.283842] alg: No test for adler32 (adler32-zlib) [10617.349736] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [10617.989568] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10619.807252] Key type lgssc registered [10621.125790] Lustre: Echo OBD driver; http://www.lustre.org/ [10629.282956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10629.297142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10630.898334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10635.907968] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10640.355954] LustreError: 322141:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10640.363893] LustreError: 322141:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10645.473536] LustreError: 322140:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10647.032332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10647.690426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10653.840187] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10663.376574] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10669.726524] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10682.234041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10682.925356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10691.266250] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10701.267262] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10713.106938] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.13@tcp (stopping) [10713.820901] Lustre: server umount lustre-MDT0000 complete [10714.082354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10714.092482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10714.115696] LustreError: 322140:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.626902] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10715.661772] Lustre: Skipped 1 previous similar message [10718.203403] LustreError: 322135:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10718.237514] LustreError: 322135:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10719.878252] LustreError: 322121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851757 with bad export cookie 8029147211515213612 [10719.895950] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10719.908052] LustreError: 322121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10720.738449] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10720.756976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10723.305986] LustreError: 322137:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10723.339097] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [10723.349838] LustreError: 322137:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10726.472339] Lustre: server umount lustre-MDT0001 complete [10736.951254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10748.057021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10764.795637] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6f6d42d93faed310 [10764.814544] Lustre: MGC192.168.201.113@tcp: Connection restored to 0@lo (at 0@lo) [10765.534598] LustreError: 323972:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.570742] LustreError: 323972:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10765.744113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10765.914792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10773.559480] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10779.231722] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10784.236652] LustreError: 323972:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10784.309811] LustreError: 323972:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [10788.021785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10788.682143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10794.002672] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10795.868705] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10805.064120] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10806.845260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10813.313077] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10816.005409] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10822.112934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10822.119353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10822.149183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10822.154054] Lustre: Skipped 1 previous similar message [10824.673927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10828.467469] Lustre: server umount lustre-OST0000 complete [10834.928832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10834.932163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10834.942508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10834.942519] Lustre: Skipped 2 previous similar messages [10834.952804] Lustre: Skipped 2 previous similar messages [10845.155957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10845.160054] Lustre: Skipped 4 previous similar messages [10847.199363] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10847.334742] Lustre: server umount lustre-MDT0000 complete [10850.277230] LustreError: 325739:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.316697] LustreError: 325739:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10851.759157] LustreError: 324722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851889 with bad export cookie 8029147211515220752 [10851.781265] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10851.791054] LustreError: 324722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10852.264602] Lustre: server umount lustre-MDT0001 complete [10864.709415] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [10868.237036] Key type lgssc unregistered [10868.521140] LNet: 328416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10868.533370] LNetError: 328416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10868.575291] LNet: Removed LNI 192.168.201.113@tcp [10869.604785] Key type .llcrypt unregistered [10869.607691] Key type ._llcrypt unregistered [10896.029550] Key type ._llcrypt registered [10896.031526] Key type .llcrypt registered [10896.112217] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [10897.614501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10897.661227] alg: No test for adler32 (adler32-zlib) [10898.698875] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [10898.923219] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [10900.567684] Key type lgssc registered [10901.668391] Lustre: Echo OBD driver; http://www.lustre.org/ [10908.998484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10909.029311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10910.449495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10915.461120] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10920.931672] LustreError: 329833:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10920.951304] LustreError: 329833:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10925.743464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10926.224231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10930.779716] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10938.832885] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10945.337752] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10949.563811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10949.569737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10949.579882] Lustre: Skipped 1 previous similar message [10949.642316] Lustre: Skipped 1 previous similar message [10950.634218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10950.651027] Lustre: Skipped 1 previous similar message [10955.756894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10955.779390] Lustre: Skipped 1 previous similar message [10956.365239] Lustre: server umount lustre-MDT0000 complete [10960.867816] LustreError: 329833:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10960.908623] LustreError: 329833:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10963.572690] LustreError: 329814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852001 with bad export cookie 9999363804903570342 [10963.591717] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10964.212731] Lustre: server umount lustre-MDT0001 complete [10983.401915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.077502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10984.132756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10984.336917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10989.531736] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [10998.976327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10999.069639] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10999.105585] Lustre: 332421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10999.488916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11005.330050] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11013.587574] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11019.760844] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11029.882894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11030.216217] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11030.231883] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11030.248629] LustreError: 334307:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [11030.259837] LustreError: 334307:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [11030.266396] LustreError: 334307:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [11030.377530] Lustre: server umount lustre-OST0000 complete [11030.380435] LustreError: 334307:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [11041.456324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11041.737825] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11042.038463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11048.866184] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11051.691712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [11057.593588] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11065.943247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11066.091434] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11066.252854] Lustre: lustre-OST0001: new disk, initializing [11066.263890] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11066.270957] Lustre: 335852:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11066.446618] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11074.515171] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11076.260961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11076.267547] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11076.431618] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11084.344648] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11091.943031] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11091.944209] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11091.946498] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11091.950776] LustreError: Skipped 1 previous similar message [11091.996280] Lustre: Skipped 1 previous similar message [11095.130701] Lustre: server umount lustre-OST0001 complete [11097.105594] LustreError: 334775:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11097.113810] LustreError: 334775:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11102.183745] LustreError: 334776:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.208569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11102.220846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11102.220862] Lustre: Skipped 1 previous similar message [11102.247538] LustreError: 334776:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11102.293263] Lustre: Skipped 1 previous similar message [11106.282484] Lustre: server umount lustre-OST0000 complete [11111.908789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11111.919374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11111.951547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11111.960021] Lustre: Skipped 1 previous similar message [11125.215226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11125.483140] Lustre: server umount lustre-MDT0000 complete [11127.779362] LustreError: 332434:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11127.807310] LustreError: 332434:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11129.714580] LustreError: 332416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852167 with bad export cookie 9999363804903570895 [11129.718090] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11129.732296] LustreError: 332416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11130.278531] Lustre: server umount lustre-MDT0001 complete [11141.721974] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11145.291331] Key type lgssc unregistered [11145.794487] LNet: 337814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11145.805810] LNetError: 337814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11145.833093] LNet: Removed LNI 192.168.201.113@tcp [11147.013910] Key type .llcrypt unregistered [11147.016265] Key type ._llcrypt unregistered [11241.263958] Key type ._llcrypt registered [11241.265187] Key type .llcrypt registered [11241.399972] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [11257.326423] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11259.206034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11259.331422] alg: No test for adler32 (adler32-zlib) [11260.565940] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [11261.075240] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11262.831762] Key type lgssc registered [11264.357640] Lustre: Echo OBD driver; http://www.lustre.org/ [11304.833313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11304.890241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11306.055536] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11306.081957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11306.154352] Lustre: lustre-MDT0000: new disk, initializing [11306.231536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11306.245856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11311.006737] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11325.550589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11325.773434] Lustre: 342750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11325.884087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11325.889555] Lustre: Skipped 1 previous similar message [11326.077297] Lustre: lustre-MDT0001: new disk, initializing [11326.242408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11326.285210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11326.293634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11333.453935] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11346.511100] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11353.057295] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11361.433374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11361.768953] Lustre: lustre-OST0000: new disk, initializing [11361.772660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11361.787584] Lustre: 344722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11361.890469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11368.634614] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11369.055078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11369.074506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11369.202970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11378.827882] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11384.197702] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11384.521473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11389.135200] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11389.328255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11395.048311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11395.055918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11395.057776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11395.083024] Lustre: Skipped 1 previous similar message [11397.534567] Lustre: server umount lustre-OST0000 complete [11403.237530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11403.244960] LustreError: Skipped 1 previous similar message [11403.252581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11403.266177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11403.279319] Lustre: Skipped 1 previous similar message [11405.280458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11405.283228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11407.818763] Lustre: server umount lustre-MDT0000 complete [11411.681945] LustreError: 343549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852449 with bad export cookie 489331267208298107 [11411.698352] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11411.699876] LustreError: 343549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11412.058658] Lustre: server umount lustre-MDT0001 complete [11421.326676] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:40:57 (1787852457) [11430.449819] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:41:06 (1787852466) [11441.247044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11441.925976] LustreError: 347480:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11441.951962] LustreError: 347480:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11442.013824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11447.267712] LustreError: 347481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11448.515607] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11451.376708] LustreError: 347480:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11456.489498] LustreError: 347481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11458.829161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11459.174234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11464.509986] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11471.797898] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11477.971323] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11484.961576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11485.250992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11491.663553] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11500.242682] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11516.404631] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11520.068809] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11523.014664] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11526.464424] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11529.952688] Lustre: Failing over lustre-MDT0000 [11530.405782] Lustre: server umount lustre-MDT0000 complete [11530.725380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11530.748332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.786764] LustreError: 348208:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11539.427130] LustreError: 347481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11539.448732] LustreError: 347481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [11550.560100] Lustre: 340978:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852572/real 1787852572] req@00000000ba089a08 x1874699011307904/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787852588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11550.592729] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11551.288941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11555.809905] LustreError: 347481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11555.824574] LustreError: 347481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [11561.270538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11561.325191] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11566.060912] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11566.584063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11566.625685] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11566.694678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11567.910443] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11585.106764] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11588.113501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11592.058861] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11594.254510] Lustre: Failing over lustre-MDT0000 [11594.787181] Lustre: server umount lustre-MDT0000 complete [11596.825986] LustreError: 349995:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11596.844576] LustreError: 349995:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [11597.285493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11597.287474] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11597.326612] Lustre: Skipped 4 previous similar messages [11613.663191] Lustre: 340978:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787852635/real 1787852635] req@00000000020bd33b x1874699011333120/t0(0) o400->MGC192.168.201.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787852651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11613.690052] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11614.511503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11623.906357] LustreError: 340974:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b5512429 x1874699011339136/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11624.356650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11624.413970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11627.498928] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11629.561164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11629.565431] Lustre: Skipped 2 previous similar messages [11629.607090] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11629.709988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11630.648585] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11646.422016] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11649.769806] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11660.259291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11660.282651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11660.294353] Lustre: Skipped 2 previous similar messages [11660.296893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11660.341852] Lustre: Skipped 3 previous similar messages [11663.003091] Lustre: server umount lustre-MDT0000 complete [11665.395693] LustreError: 349995:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11665.432670] LustreError: 349995:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 28 previous similar messages [11667.585567] LustreError: 347462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852705 with bad export cookie 489331267208300823 [11667.601324] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11667.603096] LustreError: 347462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11667.976323] Lustre: server umount lustre-MDT0001 complete [11683.204580] Lustre: server umount lustre-OST0000 complete [11704.036216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11704.732741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11709.521292] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11718.185729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11722.852333] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11730.937286] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11737.985235] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11745.961940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11749.718166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11756.586793] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11766.380795] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11774.440553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11774.442824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11774.463557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11774.470835] Lustre: Skipped 1 previous similar message [11777.436406] Lustre: server umount lustre-OST0000 complete [11784.674453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11788.059439] Lustre: server umount lustre-MDT0000 complete [11792.082382] LustreError: 355698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852829 with bad export cookie 489331267208301565 [11792.090524] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11792.099554] LustreError: 355698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11792.572325] Lustre: server umount lustre-MDT0001 complete [11808.058177] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [11811.940896] Key type lgssc unregistered [11812.381590] LNet: 357391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11812.390781] LNetError: 357391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11812.423720] LNet: Removed LNI 192.168.201.113@tcp [11813.792352] Key type .llcrypt unregistered [11813.804828] Key type ._llcrypt unregistered [11845.447324] Key type ._llcrypt registered [11845.451434] Key type .llcrypt registered [11845.571160] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [11846.762149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11846.791583] alg: No test for adler32 (adler32-zlib) [11847.917768] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [11848.284360] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [11849.999114] Key type lgssc registered [11851.648919] Lustre: Echo OBD driver; http://www.lustre.org/ [11863.270501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11863.301095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11864.892345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11871.093441] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11874.278916] LustreError: 358910:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11874.337108] LustreError: 358910:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11879.392738] LustreError: 358911:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11883.347181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11883.985692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11889.680918] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11897.909902] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11905.487696] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11913.260308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11913.842768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11921.210833] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11929.878750] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11935.361823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11945.953912] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11945.961380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11945.965178] Lustre: Skipped 1 previous similar message [11950.560470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11950.572769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11950.588072] Lustre: Skipped 1 previous similar message [11950.603090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11952.130480] Lustre: server umount lustre-MDT0000 complete [11955.951492] LustreError: 358891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852993 with bad export cookie 1112221089294773911 [11955.963663] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11956.241164] Lustre: server umount lustre-MDT0001 complete [11971.087519] Lustre: server umount lustre-OST0000 complete [11986.539328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11987.050915] LustreError: 362648:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11987.134042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11992.557297] LustreError: 362649:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11992.907527] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [11996.648680] LustreError: 362648:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12003.526912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12003.928341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12009.501352] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12018.306172] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12024.794273] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12032.777983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12033.397784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12041.293548] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12050.744315] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12056.953851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [12075.489993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12075.500589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12075.524398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12075.527043] Lustre: Skipped 3 previous similar messages [12077.541132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12077.542114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12077.552626] Lustre: Skipped 1 previous similar message [12077.560463] Lustre: Skipped 1 previous similar message [12081.693694] Lustre: server umount lustre-MDT0000 complete [12082.657760] LustreError: 362648:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.698432] LustreError: 362648:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12085.472793] LustreError: 364492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853123 with bad export cookie 1112221089294775024 [12085.484581] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12085.487940] LustreError: 364492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12086.001816] Lustre: server umount lustre-MDT0001 complete [12100.746065] Lustre: server umount lustre-OST0000 complete [12119.744210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12120.157813] LustreError: 366585:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12120.245406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12125.861627] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12136.751395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12143.441931] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12153.151458] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12160.464552] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12167.878401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12168.313650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12168.316652] Lustre: Skipped 1 previous similar message [12175.947316] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12184.780590] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12190.911580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [12202.476607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12202.484759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.505680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12206.590382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12208.661554] Lustre: server umount lustre-OST0000 complete [12214.241803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12214.258815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12214.284683] Lustre: Skipped 1 previous similar message [12214.302696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12214.308131] Lustre: Skipped 2 previous similar messages [12220.983132] Lustre: server umount lustre-MDT0000 complete [12221.929147] LustreError: 366580:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12221.976513] LustreError: 366580:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12225.709922] LustreError: 366565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853263 with bad export cookie 1112221089294776046 [12225.756562] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12226.201813] Lustre: server umount lustre-MDT0001 complete [12238.481252] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12242.142522] Key type lgssc unregistered [12242.434110] LNet: 370146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12242.438768] LNetError: 370146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12242.475683] LNet: Removed LNI 192.168.201.113@tcp [12243.626117] Key type .llcrypt unregistered [12243.629078] Key type ._llcrypt unregistered [12273.257498] Key type ._llcrypt registered [12273.260428] Key type .llcrypt registered [12273.346286] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12274.325924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12274.430090] alg: No test for adler32 (adler32-zlib) [12275.569532] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [12275.841729] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12277.519213] Key type lgssc registered [12278.847642] Lustre: Echo OBD driver; http://www.lustre.org/ [12287.711754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12287.738734] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12287.752562] LustreError: 371588:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12287.775212] LustreError: 371588:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12287.779867] LustreError: 371588:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12287.786115] LustreError: 371588:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12287.792233] LustreError: 371588:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12297.254660] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12297.266279] LustreError: 371985:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12297.290416] LustreError: 371985:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12297.312492] LustreError: 371985:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12297.322637] LustreError: 371985:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12297.337466] LustreError: 371985:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12320.255288] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12323.944991] Key type lgssc unregistered [12324.232856] LNet: 372991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12324.238897] LNetError: 372991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12324.252393] LNet: Removed LNI 192.168.201.113@tcp [12325.233634] Key type .llcrypt unregistered [12325.239368] Key type ._llcrypt unregistered [12348.248462] Key type ._llcrypt registered [12348.250752] Key type .llcrypt registered [12348.391481] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [12365.205818] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12366.493262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12366.579188] alg: No test for adler32 (adler32-zlib) [12367.761075] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [12368.053552] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12369.703600] Key type lgssc registered [12371.148247] Lustre: Echo OBD driver; http://www.lustre.org/ [12408.890624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12408.911589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12410.268944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12410.328984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12410.467691] Lustre: lustre-MDT0000: new disk, initializing [12410.573351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12410.591693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12415.587541] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12426.725736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.882991] Lustre: 376148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12426.952872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12426.960093] Lustre: Skipped 1 previous similar message [12427.050678] Lustre: lustre-MDT0001: new disk, initializing [12427.137050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12427.179034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12427.191633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12432.154488] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12443.244486] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12449.633814] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12458.882633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12459.262949] Lustre: lustre-OST0000: new disk, initializing [12459.278076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12459.288046] Lustre: 378122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12459.403119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12465.246437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12465.266440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12465.418276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12467.842145] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12478.325180] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12483.733685] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12483.968225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12489.123891] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12489.486968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12496.354525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12496.358327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12496.367828] LustreError: Skipped 1 previous similar message [12496.372245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12496.384863] Lustre: Skipped 1 previous similar message [12501.478656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12501.484974] Lustre: Skipped 1 previous similar message [12506.081230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12506.363460] Lustre: server umount lustre-OST0000 complete [12511.714317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12511.735942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12511.746207] Lustre: Skipped 2 previous similar messages [12516.827323] Lustre: server umount lustre-MDT0000 complete [12521.877770] LustreError: 376142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853559 with bad export cookie 13929369228372567872 [12521.882825] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12522.588749] Lustre: server umount lustre-MDT0001 complete [12536.711338] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12539.570908] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:59:35 (1787853575) [12547.640169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12548.200975] LustreError: 380585:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12548.217360] LustreError: 380585:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12548.275074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12553.703470] LustreError: 380586:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12554.084484] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12558.819091] LustreError: 380585:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12563.977567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12564.346046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12568.788241] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12577.804124] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12583.995668] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12591.692693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12592.051435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12597.886755] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12605.979401] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12613.669580] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12613.824503] Lustre: lustre-OST0001: new disk, initializing [12613.828111] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12613.834142] Lustre: 383425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12613.929131] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12619.884962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12619.894315] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12619.980112] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12620.710216] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12630.876486] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12635.629626] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12635.648309] Lustre: Skipped 1 previous similar message [12635.668971] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12635.670186] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12635.691574] Lustre: Skipped 1 previous similar message [12640.754045] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12640.763732] Lustre: Skipped 2 previous similar messages [12641.115339] Lustre: server umount lustre-OST0001 complete [12642.806046] LustreError: 382410:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12642.820460] LustreError: 382410:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12650.976988] LustreError: 382411:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12650.998105] LustreError: 382411:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12660.705114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12660.714127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12660.726104] Lustre: Skipped 1 previous similar message [12660.734692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12661.239362] LustreError: 382409:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12661.260087] LustreError: 382409:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12666.494174] Lustre: server umount lustre-OST0000 complete [12671.462587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12671.483548] Lustre: Skipped 2 previous similar messages [12677.076784] Lustre: server umount lustre-MDT0000 complete [12681.130805] LustreError: 380566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853719 with bad export cookie 13929369228372568747 [12681.133923] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12681.141908] LustreError: 380566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12687.581811] Lustre: server umount lustre-MDT0001 complete [12699.177281] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [12702.082380] Key type lgssc unregistered [12702.374814] LNet: 385424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12702.386793] LNetError: 385424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12702.403366] LNet: Removed LNI 192.168.201.113@tcp [12703.187582] Key type .llcrypt unregistered [12703.189909] Key type ._llcrypt unregistered [12759.576414] Key type ._llcrypt registered [12759.578655] Key type .llcrypt registered [12759.797482] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [12761.546985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12761.561932] alg: No test for adler32 (adler32-zlib) [12762.589515] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [12762.790119] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [12764.440471] Key type lgssc registered [12765.744957] Lustre: Echo OBD driver; http://www.lustre.org/ [12776.591196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12776.656610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12778.264532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12778.335442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12778.555544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12785.611392] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12797.749843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12797.885121] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12797.949085] Lustre: 388226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12798.324890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12803.960782] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12812.888102] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12819.314949] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12824.036168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12824.038161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12824.041962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12824.054265] Lustre: Skipped 1 previous similar message [12828.744415] Lustre: server umount lustre-MDT0000 complete [12833.291758] LustreError: 388218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853871 with bad export cookie 12704368571079840425 [12833.296835] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12833.314589] LustreError: 388218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12833.838333] Lustre: server umount lustre-MDT0001 complete [12843.897912] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:04:40 (1787853880) [12851.407552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12851.894404] LustreError: 390739:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.922061] LustreError: 390739:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12851.978267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12856.759318] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12857.321673] LustreError: 390740:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12862.434102] LustreError: 390739:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12864.651031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12864.940766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12868.651265] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12875.840849] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12882.128332] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12889.357772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12889.578183] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12889.722217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12895.466939] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12902.976128] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12911.496855] Lustre: Permanently deactivating lustre-OST0000 [12911.510050] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12917.283242] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12919.903701] LustreError: 393675:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12919.918079] LustreError: 393675:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12919.925192] LustreError: 393675:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12922.924381] LustreError: 393724:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12922.930450] LustreError: 393724:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12931.446726] Lustre: server umount lustre-OST0000 complete [12934.667389] LustreError: 393973:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12934.680330] LustreError: 393973:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12938.725083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12938.733472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12938.745581] Lustre: Skipped 1 previous similar message [12938.750869] Lustre: Skipped 1 previous similar message [12943.841442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12943.850422] Lustre: Skipped 2 previous similar messages [12947.008687] Lustre: server umount lustre-MDT0000 complete [12948.970180] LustreError: 390735:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12948.992587] LustreError: 390735:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12951.051665] LustreError: 392581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853988 with bad export cookie 12704368571079840999 [12951.060215] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12951.070260] LustreError: 392581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12960.483060] Lustre: server umount lustre-MDT0001 complete [12972.633391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12978.069446] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [12981.344421] LustreError: 395134:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12988.377726] Lustre: 395473:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.113@tcp [12991.059878] Lustre: server umount MGS complete [13003.149223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13003.539175] LustreError: 396197:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13003.554709] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13003.566165] Lustre: Skipped 1 previous similar message [13003.616420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13008.402292] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13013.989117] LustreError: 396198:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13014.001836] LustreError: 396198:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13017.582934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13018.154128] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [13023.742331] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13032.034360] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13038.028234] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13040.093295] Lustre: Permanently reactivating lustre-OST0000 [13040.107450] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [13040.114185] Lustre: Skipped 2 previous similar messages [13069.793151] LustreError: 398026:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [13077.861422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13078.319952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13078.323021] Lustre: Skipped 1 previous similar message [13085.202395] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13093.877963] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13100.021895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13100.025093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13100.039884] LustreError: Skipped 1 previous similar message [13100.068446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13100.080445] Lustre: Skipped 1 previous similar message [13100.871378] Lustre: server umount lustre-OST0000 complete [13105.129704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13105.136158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13105.151662] Lustre: Skipped 1 previous similar message [13105.158062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13110.721418] Lustre: server umount lustre-MDT0000 complete [13113.314914] LustreError: 396194:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13113.330686] LustreError: 396194:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13114.007026] LustreError: 396177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854151 with bad export cookie 12704368571079842056 [13114.009370] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13114.014888] LustreError: 396177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13114.271599] Lustre: server umount lustre-MDT0001 complete [13124.398818] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13127.035331] Key type lgssc unregistered [13127.331279] LNet: 400612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13127.348458] LNetError: 400612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13127.366798] LNet: Removed LNI 192.168.201.113@tcp [13128.526204] Key type .llcrypt unregistered [13128.528343] Key type ._llcrypt unregistered [13151.745693] Key type ._llcrypt registered [13151.753055] Key type .llcrypt registered [13152.035592] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [13168.767352] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13170.370369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13170.405512] alg: No test for adler32 (adler32-zlib) [13171.430892] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [13171.705424] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13173.359854] Key type lgssc registered [13174.570731] Lustre: Echo OBD driver; http://www.lustre.org/ [13208.891084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13208.928343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13210.347267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13210.384806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13210.453624] Lustre: lustre-MDT0000: new disk, initializing [13210.534753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13210.547377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13215.344108] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13226.935282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13227.091727] Lustre: 403772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13227.150276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13227.160104] Lustre: Skipped 1 previous similar message [13227.248306] Lustre: lustre-MDT0001: new disk, initializing [13227.367112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13227.400879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13227.414872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13232.362475] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13243.582748] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13250.948111] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13259.067394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13259.296735] Lustre: lustre-OST0000: new disk, initializing [13259.299112] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13259.302692] Lustre: 405739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13259.344135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13267.709345] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13268.131795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13268.178458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13268.414604] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13280.080693] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13284.550487] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13284.757636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13289.777466] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13290.163402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13294.049660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13294.049753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13294.073832] LustreError: Skipped 1 previous similar message [13294.076798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13294.092182] Lustre: Skipped 1 previous similar message [13298.751480] Lustre: server umount lustre-OST0000 complete [13304.293074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13304.298899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13304.312494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13304.334056] Lustre: Skipped 2 previous similar messages [13308.927150] Lustre: server umount lustre-MDT0000 complete [13313.070928] LustreError: 403763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854350 with bad export cookie 16279009819929848480 [13313.079256] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13313.091775] LustreError: 403763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13313.466957] Lustre: server umount lustre-MDT0001 complete [13324.431198] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13326.450734] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:12:42 (1787854362) [13334.932525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13335.389117] LustreError: 408199:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13335.406198] LustreError: 408199:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13335.457931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13340.221514] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13340.641176] LustreError: 408200:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13345.770573] LustreError: 408199:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13349.285744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13349.769761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13354.504761] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13362.572607] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13368.729817] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13376.076698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13376.535745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13382.586662] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13389.770101] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13391.700677] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13393.892686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13405.152415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13405.165182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13405.176403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13405.184077] Lustre: Skipped 2 previous similar messages [13405.200020] Lustre: Skipped 1 previous similar message [13409.789823] Lustre: server umount lustre-OST0000 complete [13415.398240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13415.416163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13415.418868] Lustre: Skipped 1 previous similar message [13415.447972] Lustre: Skipped 1 previous similar message [13420.395263] Lustre: server umount lustre-MDT0000 complete [13420.512757] LustreError: 408200:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13420.532961] LustreError: 408200:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13423.975285] LustreError: 408934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854461 with bad export cookie 16279009819929849362 [13423.977265] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13423.984811] LustreError: 408934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13424.293880] Lustre: server umount lustre-MDT0001 complete [13434.746168] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13437.363862] Key type lgssc unregistered [13437.687094] LNet: 411928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13437.695376] LNetError: 411928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13438.767774] LNet: Removed LNI 192.168.201.113@tcp [13440.040557] Key type .llcrypt unregistered [13440.044873] Key type ._llcrypt unregistered [13465.778586] Key type ._llcrypt registered [13465.781294] Key type .llcrypt registered [13465.874499] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13466.229164] Key type .llcrypt unregistered [13466.232710] Key type ._llcrypt unregistered [13484.898445] Key type ._llcrypt registered [13484.901254] Key type .llcrypt registered [13485.044536] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13486.401047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13486.505816] alg: No test for adler32 (adler32-zlib) [13487.732968] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [13488.068710] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13489.730038] Key type lgssc registered [13491.174715] Lustre: Echo OBD driver; http://www.lustre.org/ [13498.786538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13498.798530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13500.169045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13504.698877] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13509.609459] LustreError: 414148:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13509.627296] LustreError: 414148:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13512.814985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13513.294689] LustreError: 414147:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13513.309194] LustreError: 414147:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13513.314487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13518.308310] LustreError: 414143:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13518.326926] LustreError: 414143:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13519.343609] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13523.431545] LustreError: 414148:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13523.459120] LustreError: 414148:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13524.472102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13527.771385] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13529.579712] LustreError: 414143:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13529.598025] LustreError: 414143:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13535.418033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13535.809435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13539.932911] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13550.048645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13550.055633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13550.071223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13551.100715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13551.101878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13551.115438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13551.865847] Lustre: server umount lustre-OST0000 complete [13556.198227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13556.202200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13556.232388] Lustre: Skipped 1 previous similar message [13556.236200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13556.248550] Lustre: Skipped 2 previous similar messages [13561.316824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13562.830078] Lustre: server umount lustre-MDT0000 complete [13566.432752] LustreError: 414147:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13566.454807] LustreError: 414147:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13566.733936] LustreError: 414129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854604 with bad export cookie 11319693449988752222 [13566.746572] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13566.750057] LustreError: 414129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13567.213663] Lustre: server umount lustre-MDT0001 complete [13579.655510] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13582.170289] Key type lgssc unregistered [13582.395973] LNet: 417300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13582.412325] LNetError: 417300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13582.432495] LNet: Removed LNI 192.168.201.113@tcp [13583.409455] Key type .llcrypt unregistered [13583.417379] Key type ._llcrypt unregistered [13607.792318] Key type ._llcrypt registered [13607.798575] Key type .llcrypt registered [13607.933698] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13609.471953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13609.526657] alg: No test for adler32 (adler32-zlib) [13610.638864] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [13610.875467] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13612.599177] Key type lgssc registered [13613.668104] Lustre: Echo OBD driver; http://www.lustre.org/ [13623.223484] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13623.257695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13638.815377] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13643.935373] LustreError: 418690:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.113@tcp: failed processing log, type 4: rc = -110 [13670.559256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13676.287509] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13682.872499] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13689.899707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13690.525711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13690.599828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13695.466623] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13695.969101] LustreError: 418716:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13695.983060] LustreError: 418716:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13701.094289] LustreError: 418715:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13703.529282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13703.816505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13708.965754] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13716.572223] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13722.638172] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13729.251838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13729.257062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13729.278606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13730.887700] Lustre: server umount lustre-OST0000 complete [13736.932841] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13736.932949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13736.937945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13736.955059] Lustre: Skipped 1 previous similar message [13742.048176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13742.058282] Lustre: Skipped 2 previous similar messages [13745.262590] Lustre: server umount lustre-MDT0000 complete [13746.145116] LustreError: 419731:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13746.162262] LustreError: 419731:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13748.404085] LustreError: 418696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854786 with bad export cookie 10926500782996258293 [13748.409961] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13748.422481] LustreError: 418696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13748.640337] Lustre: server umount lustre-MDT0001 complete [13760.214750] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13762.846253] Key type lgssc unregistered [13763.187977] LNet: 422261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13763.194100] LNetError: 422261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13763.212697] LNet: Removed LNI 192.168.201.113@tcp [13764.024243] Key type .llcrypt unregistered [13764.032827] Key type ._llcrypt unregistered [13790.585836] Key type ._llcrypt registered [13790.588694] Key type .llcrypt registered [13790.727807] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13791.808652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13791.880461] alg: No test for adler32 (adler32-zlib) [13792.985063] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [13793.345309] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13795.096928] Key type lgssc registered [13796.304070] Lustre: Echo OBD driver; http://www.lustre.org/ [13804.758820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13804.785901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13806.286181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13811.450088] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13816.807833] LustreError: 423678:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13816.815075] LustreError: 423678:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13820.294255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13820.604401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13825.104284] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13832.789823] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13838.560397] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13847.486024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13847.940756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13852.051859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13855.889763] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13867.784230] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13874.980556] Lustre: server umount lustre-MDT0000 complete [13875.202033] LustreError: 423674:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13876.715464] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13876.720791] LustreError: 423675:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13876.746227] Lustre: Skipped 1 previous similar message [13877.833926] LustreError: 423660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.13@tcp arrived at 1787854915 with bad export cookie 13489256113491326168 [13881.021529] LustreError: 423659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854918 with bad export cookie 13489256113491325272 [13881.041878] LustreError: 423659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13881.055472] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13881.827524] LustreError: 423678:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13881.828605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13881.828735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13881.830164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13881.889928] LustreError: 423678:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13886.945631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13886.946963] LustreError: 424725:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13886.970226] LustreError: 424725:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13887.358861] Lustre: server umount lustre-OST0000 complete [13893.892408] Lustre: server umount lustre-MDT0001 complete [13905.424401] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [13907.905737] Key type lgssc unregistered [13908.129686] LNet: 427363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13908.140412] LNetError: 427363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13908.155171] LNet: Removed LNI 192.168.201.113@tcp [13909.055376] Key type .llcrypt unregistered [13909.058550] Key type ._llcrypt unregistered [13934.312944] Key type ._llcrypt registered [13934.315641] Key type .llcrypt registered [13934.433471] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [13935.446959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13935.581146] alg: No test for adler32 (adler32-zlib) [13936.644556] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [13936.899450] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [13938.527174] Key type lgssc registered [13939.589430] Lustre: Echo OBD driver; http://www.lustre.org/ [13947.421321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13947.450715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13949.005108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13953.411491] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13959.141089] LustreError: 428785:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13959.175533] LustreError: 428785:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13962.028314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13962.602696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13967.379271] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [13975.582500] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13981.908273] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13989.494533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13989.804974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13994.929982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13995.694488] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14001.868512] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14008.801327] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14008.808165] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14008.819060] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14009.833559] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [14010.355198] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14012.916503] Lustre: server umount lustre-MDT0001 complete [14015.461149] LustreError: 428781:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14015.477165] LustreError: 428781:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14020.578673] LustreError: 429512:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14020.579365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14020.581033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14020.582861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14020.582868] Lustre: Skipped 2 previous similar messages [14025.703486] LustreError: 428784:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14025.779633] Lustre: server umount lustre-OST0000 complete [14030.191981] Lustre: server umount lustre-MDT0000 complete [14044.766967] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [14047.465934] Key type lgssc unregistered [14047.708641] LNet: 432468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14047.722698] LNetError: 432468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14047.749653] LNet: Removed LNI 192.168.201.113@tcp [14048.590618] Key type .llcrypt unregistered [14048.593742] Key type ._llcrypt unregistered [14074.837742] Key type ._llcrypt registered [14074.842952] Key type .llcrypt registered [14074.929509] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [14089.518432] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [14090.550137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14090.560602] alg: No test for adler32 (adler32-zlib) [14091.615163] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [14091.930613] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [14093.647146] Key type lgssc registered [14094.785827] Lustre: Echo OBD driver; http://www.lustre.org/ [14132.577417] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [14152.575939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14152.620930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14154.120691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14154.187636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14154.341412] Lustre: lustre-MDT0000: new disk, initializing [14154.464274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14154.512603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14160.633530] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14178.564764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14179.026454] Lustre: lustre-OST0000: new disk, initializing [14179.028505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14179.032160] Lustre: Skipped 1 previous similar message [14179.035554] Lustre: 437506:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14179.109198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14187.580347] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14189.170851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14189.198792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14189.228989] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14198.838740] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14214.697731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14214.816576] Lustre: 436485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14214.907628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14214.978753] Lustre: lustre-MDT0001: new disk, initializing [14215.127360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14215.204039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14215.219139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14221.955160] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14236.922157] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14237.351028] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14243.132350] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14243.488231] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14250.467077] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14250.480855] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14250.520166] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14251.037612] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.13@tcp (stopping) [14251.054699] Lustre: Skipped 1 previous similar message [14255.590443] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14255.601309] Lustre: Skipped 1 previous similar message [14260.713304] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14260.733352] Lustre: Skipped 2 previous similar messages [14263.263148] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14263.637868] Lustre: server umount lustre-MDT0001 complete [14270.962142] LustreError: 436498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14270.985212] LustreError: 436498:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [14276.074921] LustreError: 436499:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14276.113601] LustreError: 436499:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14281.190624] LustreError: 436494:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14281.203358] LustreError: 436494:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14286.305413] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14286.307567] LustreError: 436499:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14286.320494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14286.320502] Lustre: Skipped 1 previous similar message [14286.324818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14286.403392] Lustre: Skipped 1 previous similar message [14287.928316] Lustre: server umount lustre-OST0000 complete [14291.426623] LustreError: 436498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14306.271142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14306.623578] Lustre: server umount lustre-MDT0000 complete [14322.494710] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [14325.604914] Key type lgssc unregistered [14325.883706] LNet: 441259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14325.893909] LNetError: 441259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14325.907368] LNet: Removed LNI 192.168.201.113@tcp [14326.957578] Key type .llcrypt unregistered [14326.959892] Key type ._llcrypt unregistered [14372.127952] Key type ._llcrypt registered [14372.133062] Key type .llcrypt registered [14372.301569] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [14372.809591] Key type .llcrypt unregistered [14372.814660] Key type ._llcrypt unregistered [14394.390993] Key type ._llcrypt registered [14394.394617] Key type .llcrypt registered [14394.470187] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [14395.088944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14395.187733] alg: No test for adler32 (adler32-zlib) [14396.304125] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [14396.594639] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [14398.329081] Key type lgssc registered [14400.327933] Lustre: Echo OBD driver; http://www.lustre.org/ [14434.050557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14434.080897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14435.368865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14435.418733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14435.506775] Lustre: lustre-MDT0000: new disk, initializing [14435.584880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14435.604513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14440.591689] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14451.336203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14451.560597] Lustre: 444642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14451.698706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14451.711297] Lustre: Skipped 1 previous similar message [14451.849209] Lustre: lustre-MDT0001: new disk, initializing [14452.032548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14452.119770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14452.138487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14457.037527] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14469.200482] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14476.896602] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14484.272899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14484.746203] Lustre: lustre-OST0000: new disk, initializing [14484.758205] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14484.778334] Lustre: 446611:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14484.958393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14490.729680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14490.744722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14490.904526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14492.681559] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14502.277885] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14511.585887] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14511.594350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14511.610498] Lustre: Skipped 1 previous similar message [14511.626407] Lustre: Skipped 2 previous similar messages [14515.764410] Lustre: server umount lustre-MDT0000 complete [14519.795050] LustreError: 444635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855557 with bad export cookie 8138244561734405261 [14519.811036] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14519.817984] LustreError: 444635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14520.284787] Lustre: server umount lustre-MDT0001 complete [14534.645663] Lustre: server umount lustre-OST0000 complete [14546.426467] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:33:02 (1787855582) [14555.587171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14556.060616] LustreError: 448640:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14556.083271] LustreError: 448640:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14556.155536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14560.974108] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14561.269378] LustreError: 448641:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14565.349128] LustreError: 448640:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14570.471431] LustreError: 448641:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14572.408371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14572.991104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14579.634976] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14588.979956] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14597.916398] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14618.486928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14618.681980] Lustre: Found index 0 for lustre-OST0000, updating log [14618.892088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14624.970259] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [14626.107743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14632.526411] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14639.766140] Lustre: server umount lustre-OST0000 complete [14658.594609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14658.597670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14658.617687] Lustre: Skipped 2 previous similar messages [14659.699697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14659.708968] Lustre: Skipped 2 previous similar messages [14665.186275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14666.337584] Lustre: server umount lustre-MDT0000 complete [14670.313119] LustreError: 448640:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14670.336835] LustreError: 448640:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14670.380825] LustreError: 448620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855708 with bad export cookie 8138244561734406654 [14670.408446] LustreError: 448620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14670.412907] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14670.975960] Lustre: server umount lustre-MDT0001 complete [14680.470576] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:35:16 (1787855716) [14685.471926] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [15308.062124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15308.699368] LustreError: 455052:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15308.820071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15313.603527] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15313.898303] LustreError: 455053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15319.010651] LustreError: 455052:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15322.129773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15322.628070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15327.451958] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15336.787994] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15342.933270] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15364.282317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15364.395502] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [15364.598397] Lustre: Found index 0 for lustre-OST0000, updating log [15365.142329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15372.325385] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15381.584421] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15385.250791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15385.291677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15385.299360] Lustre: Skipped 1 previous similar message [15390.451405] Lustre: server umount lustre-OST0000 complete [15396.385550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15396.392291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15396.394979] Lustre: Skipped 1 previous similar message [15396.416969] Lustre: Skipped 1 previous similar message [15397.413943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15397.423677] Lustre: Skipped 1 previous similar message [15399.458351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15399.463640] Lustre: Skipped 2 previous similar messages [15401.994628] Lustre: server umount lustre-MDT0000 complete [15402.531616] LustreError: 455053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15402.556161] LustreError: 455053:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [15407.015117] LustreError: 455034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856444 with bad export cookie 8138244561734407725 [15407.038592] LustreError: 455034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15407.051364] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15407.449422] Lustre: server umount lustre-MDT0001 complete [15424.800546] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp [15440.792447] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [15451.868392] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15497.033464] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:48:53 (1787856533) [15517.000367] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15520.148182] Key type lgssc unregistered [15520.479682] LNet: 461663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15520.492563] LNetError: 461663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15628.014780] LNet: Removed LNI 192.168.201.113@tcp [15628.853557] Key type .llcrypt unregistered [15628.855785] Key type ._llcrypt unregistered [15648.752244] Key type ._llcrypt registered [15648.755469] Key type .llcrypt registered [15648.931894] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15650.777711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15650.836481] alg: No test for adler32 (adler32-zlib) [15652.143110] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [15652.451528] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15654.215263] Key type lgssc registered [15655.480652] Lustre: Echo OBD driver; http://www.lustre.org/ [15659.771185] LNet: Added LNI 192.168.201.113@tcp100 [8/256/0/180] [15660.614426] LNet: Added LNI 192.168.201.113@tcp101 [8/256/0/180] [15661.469855] LNet: Added LNI 192.168.201.113@tcp102 [8/256/0/180] [15662.464240] LNet: Added LNI 192.168.201.113@tcp103 [8/256/0/180] [15663.340266] LNet: Added LNI 192.168.201.113@tcp104 [8/256/0/180] [15664.237253] LNet: Added LNI 192.168.201.113@tcp105 [8/256/0/180] [15665.202422] LNet: Added LNI 192.168.201.113@tcp106 [8/256/0/180] [15666.366214] LNet: Added LNI 192.168.201.113@tcp107 [8/256/0/180] [15667.352857] LNet: Added LNI 192.168.201.113@tcp108 [8/256/0/180] [15668.261410] LNet: Added LNI 192.168.201.113@tcp109 [8/256/0/180] [15669.164810] LNet: Added LNI 192.168.201.113@tcp110 [8/256/0/180] [15670.208786] LNet: Added LNI 192.168.201.113@tcp111 [8/256/0/180] [15671.191929] LNet: Added LNI 192.168.201.113@tcp112 [8/256/0/180] [15672.026367] LNet: Added LNI 192.168.201.113@tcp113 [8/256/0/180] [15672.801057] LNet: Added LNI 192.168.201.113@tcp114 [8/256/0/180] [15673.716494] LNet: Added LNI 192.168.201.113@tcp115 [8/256/0/180] [15674.545126] LNet: Added LNI 192.168.201.113@tcp116 [8/256/0/180] [15675.756401] LNet: Added LNI 192.168.201.113@tcp117 [8/256/0/180] [15677.102771] LNet: Added LNI 192.168.201.113@tcp118 [8/256/0/180] [15678.343244] LNet: Added LNI 192.168.201.113@tcp119 [8/256/0/180] [15679.452573] LNet: Added LNI 192.168.201.113@tcp120 [8/256/0/180] [15680.493686] LNet: Added LNI 192.168.201.113@tcp121 [8/256/0/180] [15681.477297] LNet: Added LNI 192.168.201.113@tcp122 [8/256/0/180] [15682.434252] LNet: Added LNI 192.168.201.113@tcp123 [8/256/0/180] [15683.678099] LNet: Added LNI 192.168.201.113@tcp124 [8/256/0/180] [15684.887952] LNet: Added LNI 192.168.201.113@tcp125 [8/256/0/180] [15686.076327] LNet: Added LNI 192.168.201.113@tcp126 [8/256/0/180] [15687.240310] LNet: Added LNI 192.168.201.113@tcp127 [8/256/0/180] [15688.627910] LNet: Added LNI 192.168.201.113@tcp128 [8/256/0/180] [15689.803081] LNet: Added LNI 192.168.201.113@tcp129 [8/256/0/180] [15690.762333] LNet: Added LNI 192.168.201.113@tcp130 [8/256/0/180] [15691.732703] LNet: Added LNI 192.168.201.113@tcp131 [8/256/0/180] [15692.683821] LNet: Added LNI 192.168.201.113@tcp132 [8/256/0/180] [15693.574174] LNet: Added LNI 192.168.201.113@tcp133 [8/256/0/180] [15694.604313] LNet: Added LNI 192.168.201.113@tcp134 [8/256/0/180] [15695.532245] LNet: Added LNI 192.168.201.113@tcp135 [8/256/0/180] [15703.226192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15703.256390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15704.443391] Lustre: 464695:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [15704.553844] LNetError: 464721:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15704.587144] LNetError: Protocol error connecting to 192.168.201.113@tcp119 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15704.618325] LNetError: Skipped 9 previous similar messages [15704.673896] LNet: 7 local NIs in recovery (showing 7): 192.168.201.113@tcp125, 192.168.201.113@tcp122, 192.168.201.113@tcp128, 192.168.201.113@tcp129, 192.168.201.113@tcp127, 192.168.201.113@tcp126, 192.168.201.113@tcp119 [15704.730216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15704.770993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15704.854353] Lustre: lustre-MDT0000: new disk, initializing [15704.955757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15704.977060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15705.635432] LNetError: 464722:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15705.666989] LNetError: 464722:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [15705.684600] LNetError: Protocol error connecting to 192.168.201.113@tcp104 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15707.811760] LNetError: 462153:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15707.826856] LNetError: 462153:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [15707.845347] LNetError: Protocol error connecting to 192.168.201.113@tcp122 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15707.856912] LNetError: Skipped 6 previous similar messages [15709.839918] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15711.908509] LNetError: 462155:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15711.921776] LNetError: Protocol error connecting to 192.168.201.113@tcp122 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15719.072963] LNetError: 464716:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15719.091084] LNetError: Protocol error connecting to 192.168.201.113@tcp122 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15721.854164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15722.035568] Lustre: 464713:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15722.124370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15722.130164] Lustre: Skipped 1 previous similar message [15722.273542] Lustre: lustre-MDT0001: new disk, initializing [15722.328458] Lustre: 465514:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [15722.345920] Lustre: 465514:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15722.529169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15722.580050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15722.593615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15728.358694] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15730.726781] Lustre: 465877:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [15730.733207] Lustre: 465877:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15735.458603] LNetError: 464717:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15735.467306] LNetError: Protocol error connecting to 192.168.201.113@tcp122 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15738.591936] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15744.905743] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15752.818050] LNetError: 464717:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15752.832509] LNetError: 464717:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15752.849145] LNetError: Protocol error connecting to 192.168.201.113@tcp121 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15752.872825] LNetError: Skipped 9 previous similar messages [15753.454198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15753.833775] Lustre: lustre-OST0000: new disk, initializing [15753.840883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15753.849178] Lustre: 466690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15753.978750] Lustre: 466695:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [15753.986530] Lustre: 466695:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15754.059480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15760.502057] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15762.470947] Lustre: 467146:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.113@tcp: can't add NID 192.168.201.113@tcp130: rc = -75 [15762.482443] Lustre: 467146:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15762.569976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15762.588040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15762.718807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15765.155859] LNet: 1 local NIs in recovery (showing 1): 192.168.201.113@tcp122 [15770.104948] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15778.274782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15778.279308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15778.283239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15778.291271] Lustre: Skipped 1 previous similar message [15783.399236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15783.411272] Lustre: Skipped 2 previous similar messages [15783.772660] Lustre: server umount lustre-OST0000 complete [15788.516104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15788.518951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15788.525559] Lustre: Skipped 1 previous similar message [15793.633234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15793.638559] Lustre: Skipped 2 previous similar messages [15793.888133] Lustre: server umount lustre-MDT0000 complete [15797.858574] LNetError: 464716:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.113 [15797.873247] LNetError: 464716:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [15797.889525] LNetError: Protocol error connecting to 192.168.201.113@tcp104 at host 192.168.201.113:988: is it running a compatible version of Lustre? [15797.914276] LNetError: Skipped 18 previous similar messages [15797.935911] LustreError: 464705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856835 with bad export cookie 18376488953379991930 [15797.943298] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15797.950103] LustreError: 464705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15798.367912] Lustre: server umount lustre-MDT0001 complete [15811.931761] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15815.089306] Key type lgssc unregistered [15815.533554] LNet: 468605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15815.545401] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15815.554336] LNet: Removed LNI 192.168.201.113@tcp [15816.607182] LNet: Removed LNI 192.168.201.113@tcp100 [15816.612674] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15816.617482] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15816.640386] LNet: Removed LNI 192.168.201.113@tcp101 [15817.695176] LNet: Removed LNI 192.168.201.113@tcp102 [15817.698638] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15817.709160] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15817.725653] LNet: Removed LNI 192.168.201.113@tcp103 [15818.783990] LNet: Removed LNI 192.168.201.113@tcp104 [15819.809644] LNet: Removed LNI 192.168.201.113@tcp105 [15819.820239] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15819.832872] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15819.847621] LNet: Removed LNI 192.168.201.113@tcp106 [15820.895151] LNet: Removed LNI 192.168.201.113@tcp107 [15821.919575] LNet: Removed LNI 192.168.201.113@tcp108 [15822.943122] LNet: Removed LNI 192.168.201.113@tcp109 [15823.967378] LNet: Removed LNI 192.168.201.113@tcp110 [15823.993251] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15823.999770] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15825.055139] LNet: Removed LNI 192.168.201.113@tcp111 [15826.079574] LNet: Removed LNI 192.168.201.113@tcp112 [15827.103243] LNet: Removed LNI 192.168.201.113@tcp113 [15828.127305] LNet: Removed LNI 192.168.201.113@tcp114 [15829.151286] LNet: Removed LNI 192.168.201.113@tcp115 [15830.176492] LNet: Removed LNI 192.168.201.113@tcp116 [15831.199207] LNet: Removed LNI 192.168.201.113@tcp117 [15832.223415] LNet: Removed LNI 192.168.201.113@tcp118 [15832.225448] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15832.246814] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [15832.256140] LNet: Removed LNI 192.168.201.113@tcp119 [15833.311208] LNet: Removed LNI 192.168.201.113@tcp120 [15834.335410] LNet: Removed LNI 192.168.201.113@tcp121 [15834.344163] LNet: Removed LNI 192.168.201.113@tcp122 [15835.423354] LNet: Removed LNI 192.168.201.113@tcp123 [15836.447212] LNet: Removed LNI 192.168.201.113@tcp124 [15837.471356] LNet: Removed LNI 192.168.201.113@tcp125 [15838.495191] LNet: Removed LNI 192.168.201.113@tcp126 [15839.519201] LNet: Removed LNI 192.168.201.113@tcp127 [15840.546256] LNet: Removed LNI 192.168.201.113@tcp128 [15841.568195] LNet: Removed LNI 192.168.201.113@tcp129 [15841.575897] LNet: Removed LNI 192.168.201.113@tcp130 [15841.580282] LNet: Removed LNI 192.168.201.113@tcp131 [15841.592253] LNet: Removed LNI 192.168.201.113@tcp132 [15841.610399] LNet: Removed LNI 192.168.201.113@tcp133 [15841.622260] LNet: Removed LNI 192.168.201.113@tcp134 [15841.641763] LNet: Removed LNI 192.168.201.113@tcp135 [15842.891394] Key type .llcrypt unregistered [15842.899939] Key type ._llcrypt unregistered [15874.142388] Key type ._llcrypt registered [15874.147826] Key type .llcrypt registered [15874.360897] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [15889.919300] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15891.174670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15891.221806] alg: No test for adler32 (adler32-zlib) [15892.673641] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [15892.912431] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15894.575191] Key type lgssc registered [15895.773675] Lustre: Echo OBD driver; http://www.lustre.org/ [15931.124463] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:56:07 (1787856967) [15946.420416] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [15948.800356] Key type lgssc unregistered [15949.067723] LNet: 472697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15949.094645] LNetError: 472697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15949.121729] LNet: Removed LNI 192.168.201.113@tcp [15949.778196] Key type .llcrypt unregistered [15949.780604] Key type ._llcrypt unregistered [15964.228316] Key type ._llcrypt registered [15964.230739] Key type .llcrypt registered [15964.352106] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [15965.127252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15965.142238] alg: No test for adler32 (adler32-zlib) [15966.174571] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [15966.429761] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [15968.079319] Key type lgssc registered [15969.433681] Lustre: Echo OBD driver; http://www.lustre.org/ [15978.382291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15978.432131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15979.659563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15979.701777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15979.797140] Lustre: lustre-MDT0000: new disk, initializing [15979.873772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15979.900829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15984.131985] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [15994.713325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15994.866701] Lustre: 473890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15994.916727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15994.920333] Lustre: Skipped 1 previous similar message [15994.975851] Lustre: lustre-MDT0001: new disk, initializing [15995.046780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15995.083652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15995.089534] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16000.022408] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16008.909764] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16015.151686] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16024.147164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16024.603198] Lustre: lustre-OST0000: new disk, initializing [16024.612869] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16024.624456] Lustre: 475861:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16024.802565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16031.351612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16031.376905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16031.565360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16032.111937] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16044.367663] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16049.312170] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing lnet_if_list [16055.255340] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [16055.283598] LNet: Added LNI 192.168.201.113@tcp10 [8/256/0/180] [16061.977686] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [16062.009837] LNet: Added LNI 192.168.201.113@tcp11 [8/256/0/180] [16066.953744] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [16066.977843] LNet: Added LNI 192.168.201.113@tcp12 [8/256/0/180] [16072.488350] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [16072.523278] LNet: Added LNI 192.168.201.113@tcp13 [8/256/0/180] [16077.806453] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [16077.834983] LNet: Added LNI 192.168.201.113@tcp14 [8/256/0/180] [16083.189944] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [16083.243849] LNet: Added LNI 192.168.201.113@tcp15 [8/256/0/180] [16088.336814] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [16088.377792] LNet: Added LNI 192.168.201.113@tcp16 [8/256/0/180] [16093.938268] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [16094.007473] LNet: Added LNI 192.168.201.113@tcp17 [8/256/0/180] [16100.824335] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [16100.921179] LNet: Added LNI 192.168.201.113@tcp18 [8/256/0/180] [16108.843370] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [16108.877027] LNet: Added LNI 192.168.201.113@tcp19 [8/256/0/180] [16115.444976] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16115.490893] LNet: Added LNI 192.168.201.113@tcp20 [8/256/0/180] [16121.161870] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16121.194245] LNet: Added LNI 192.168.201.113@tcp21 [8/256/0/180] [16126.427773] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16126.454618] LNet: Added LNI 192.168.201.113@tcp22 [8/256/0/180] [16132.498821] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16132.530524] LNet: Added LNI 192.168.201.113@tcp23 [8/256/0/180] [16138.771439] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16138.796152] LNet: Added LNI 192.168.201.113@tcp24 [8/256/0/180] [16144.261353] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16144.293925] LNet: Added LNI 192.168.201.113@tcp25 [8/256/0/180] [16149.714784] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16149.771445] LNet: Added LNI 192.168.201.113@tcp26 [8/256/0/180] [16156.655504] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16156.685820] LNet: Added LNI 192.168.201.113@tcp27 [8/256/0/180] [16163.107646] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16163.137693] LNet: Added LNI 192.168.201.113@tcp28 [8/256/0/180] [16170.051921] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16170.088654] LNet: Added LNI 192.168.201.113@tcp29 [8/256/0/180] [16178.174748] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16178.210399] LNet: Added LNI 192.168.201.113@tcp30 [8/256/0/180] [16186.086436] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16186.120594] LNet: Added LNI 192.168.201.113@tcp31 [8/256/0/180] [16194.389343] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16194.418317] LNet: Added LNI 192.168.201.113@tcp32 [8/256/0/180] [16202.239591] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16202.272352] LNet: Added LNI 192.168.201.113@tcp33 [8/256/0/180] [16209.222834] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16209.250958] LNet: Added LNI 192.168.201.113@tcp34 [8/256/0/180] [16217.230023] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16217.259399] LNet: Added LNI 192.168.201.113@tcp35 [8/256/0/180] [16224.367879] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16224.414897] LNet: Added LNI 192.168.201.113@tcp36 [8/256/0/180] [16231.717507] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16231.736113] LNet: Added LNI 192.168.201.113@tcp37 [8/256/0/180] [16239.398278] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16239.436551] LNet: Added LNI 192.168.201.113@tcp38 [8/256/0/180] [16247.405165] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16247.430510] LNet: Added LNI 192.168.201.113@tcp39 [8/256/0/180] [16256.236849] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16256.265985] LNet: Added LNI 192.168.201.113@tcp40 [8/256/0/180] [16264.797338] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16264.850649] LNet: Added LNI 192.168.201.113@tcp41 [8/256/0/180] [16274.513907] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16274.537905] LNet: Added LNI 192.168.201.113@tcp42 [8/256/0/180] [16283.468538] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16283.494641] LNet: Added LNI 192.168.201.113@tcp43 [8/256/0/180] [16292.857084] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16292.896680] LNet: Added LNI 192.168.201.113@tcp44 [8/256/0/180] [16301.867724] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16301.895179] LNet: Added LNI 192.168.201.113@tcp45 [8/256/0/180] [16311.128651] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16311.181327] LNet: Added LNI 192.168.201.113@tcp46 [8/256/0/180] [16320.829135] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16320.886573] LNet: Added LNI 192.168.201.113@tcp47 [8/256/0/180] [16331.152284] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16331.198690] LNet: Added LNI 192.168.201.113@tcp48 [8/256/0/180] [16342.331962] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16342.365814] LNet: Added LNI 192.168.201.113@tcp49 [8/256/0/180] [16352.525048] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16352.557732] LNet: Added LNI 192.168.201.113@tcp50 [8/256/0/180] [16374.241407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16374.249321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16374.264125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16374.755305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16374.780424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16374.789928] Lustre: Skipped 1 previous similar message [16378.892558] Lustre: server umount lustre-MDT0000 complete [16383.434790] LustreError: 473885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857421 with bad export cookie 9420645620732310732 [16383.440903] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16383.444864] LustreError: 473885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16383.992172] Lustre: server umount lustre-MDT0001 complete [16400.351319] Lustre: 473165:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857422/real 1787857422] req@00000000e219cb67 x1874703946007168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16400.372296] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16400.378930] Lustre: Skipped 1 previous similar message [16402.558493] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16404.449179] Lustre: 473164:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857426/real 1787857426] req@00000000dd76aaed x1874703946007296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16410.601187] Lustre: 473163:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857432/real 1787857432] req@000000006d301dfc x1874703946007552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857448 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16412.412650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16415.711502] Lustre: 473162:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857437/real 1787857437] req@000000009f1ff1dd x1874703946007680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857453 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16430.111569] LustreError: 473161:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000532315fc x1874703946008448/t0(0) o250->MGC192.168.201.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16430.527371] LustreError: 475853:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16430.544844] LustreError: 475853:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [16430.610133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16431.589809] LustreError: 475854:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16435.620226] LustreError: 475853:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16436.346735] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16440.805234] LustreError: 475854:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16445.924390] LustreError: 475853:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16445.952589] LustreError: 475853:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16446.485672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16446.992423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16452.082446] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16452.093912] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16455.207567] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16459.841975] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16465.543396] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16482.786262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16482.793368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16482.803566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16485.860284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16488.257230] Lustre: server umount lustre-OST0000 complete [16493.024871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16493.042056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16493.057468] Lustre: Skipped 1 previous similar message [16493.069484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16493.072274] Lustre: Skipped 1 previous similar message [16498.488854] Lustre: server umount lustre-MDT0000 complete [16501.218266] LustreError: 500127:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16501.252099] LustreError: 500127:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16502.186244] LustreError: 473883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857540 with bad export cookie 9420645620732313434 [16502.187688] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16502.203068] LustreError: 473883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16502.743286] Lustre: server umount lustre-MDT0001 complete [16514.709914] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [16517.514438] Key type lgssc unregistered [16518.010189] LNet: 502906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16518.019726] LNetError: 502906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16518.030266] LNet: Removed LNI 192.168.201.113@tcp [16518.038054] LNet: Removed LNI 192.168.201.113@tcp10 [16518.043679] LNet: Removed LNI 192.168.201.113@tcp11 [16518.052269] LNet: Removed LNI 192.168.201.113@tcp12 [16518.054512] LNet: Removed LNI 192.168.201.113@tcp13 [16518.060314] LNet: Removed LNI 192.168.201.113@tcp14 [16518.061880] LNet: Removed LNI 192.168.201.113@tcp15 [16518.064970] LNet: Removed LNI 192.168.201.113@tcp16 [16518.067912] LNet: Removed LNI 192.168.201.113@tcp17 [16518.073111] LNet: Removed LNI 192.168.201.113@tcp18 [16518.076498] LNet: Removed LNI 192.168.201.113@tcp19 [16518.081538] LNet: Removed LNI 192.168.201.113@tcp20 [16518.085961] LNet: Removed LNI 192.168.201.113@tcp21 [16518.092365] LNet: Removed LNI 192.168.201.113@tcp22 [16518.094268] LNet: Removed LNI 192.168.201.113@tcp23 [16518.099826] LNet: Removed LNI 192.168.201.113@tcp24 [16518.105699] LNet: Removed LNI 192.168.201.113@tcp25 [16518.118214] LNet: Removed LNI 192.168.201.113@tcp26 [16518.124130] LNet: Removed LNI 192.168.201.113@tcp27 [16518.127720] LNet: Removed LNI 192.168.201.113@tcp28 [16518.138906] LNet: Removed LNI 192.168.201.113@tcp29 [16518.148629] LNet: Removed LNI 192.168.201.113@tcp30 [16518.151904] LNet: Removed LNI 192.168.201.113@tcp31 [16518.157183] LNet: Removed LNI 192.168.201.113@tcp32 [16518.171329] LNet: Removed LNI 192.168.201.113@tcp33 [16518.173694] LNet: Removed LNI 192.168.201.113@tcp34 [16518.190291] LNet: Removed LNI 192.168.201.113@tcp35 [16518.192090] LNet: Removed LNI 192.168.201.113@tcp36 [16518.201732] LNet: Removed LNI 192.168.201.113@tcp37 [16518.203772] LNet: Removed LNI 192.168.201.113@tcp38 [16518.208211] LNet: Removed LNI 192.168.201.113@tcp39 [16518.210569] LNet: Removed LNI 192.168.201.113@tcp40 [16518.214836] LNet: Removed LNI 192.168.201.113@tcp41 [16518.220847] LNet: Removed LNI 192.168.201.113@tcp42 [16518.224450] LNet: Removed LNI 192.168.201.113@tcp43 [16518.226841] LNet: Removed LNI 192.168.201.113@tcp44 [16518.228895] LNet: Removed LNI 192.168.201.113@tcp45 [16518.231266] LNet: Removed LNI 192.168.201.113@tcp46 [16518.233895] LNet: Removed LNI 192.168.201.113@tcp47 [16518.236628] LNet: Removed LNI 192.168.201.113@tcp48 [16518.238685] LNet: Removed LNI 192.168.201.113@tcp49 [16518.247991] LNet: Removed LNI 192.168.201.113@tcp50 [16518.871452] Key type .llcrypt unregistered [16518.875279] Key type ._llcrypt unregistered [16544.398649] Key type ._llcrypt registered [16544.411880] Key type .llcrypt registered [16544.578277] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [16560.580530] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16562.133452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16562.238960] alg: No test for adler32 (adler32-zlib) [16563.409282] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [16563.895902] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [16565.615161] Key type lgssc registered [16566.665645] Lustre: Echo OBD driver; http://www.lustre.org/ [16603.100687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16603.120981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16604.335333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16604.361080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16604.457342] Lustre: lustre-MDT0000: new disk, initializing [16604.545374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16604.575080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16609.288654] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16619.783407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16619.909870] Lustre: 506106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16619.994833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16619.999251] Lustre: Skipped 1 previous similar message [16620.147824] Lustre: lustre-MDT0001: new disk, initializing [16620.288980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16620.343783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16620.356267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16625.780297] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16635.741537] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16642.047749] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16650.617567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16650.930555] Lustre: *** cfs_fail_loc=90f, val=0*** [16651.052566] Lustre: lustre-OST0000: new disk, initializing [16651.056338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16651.064408] Lustre: 508126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16651.158268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16657.029192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16657.047634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16657.209905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16658.828467] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16670.919573] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16682.981900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16682.993224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16683.000147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16683.010668] Lustre: Skipped 1 previous similar message [16683.026178] Lustre: Skipped 1 previous similar message [16686.052202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16686.066304] Lustre: Skipped 1 previous similar message [16686.772632] Lustre: server umount lustre-OST0000 complete [16691.168855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16691.184663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16696.297711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16696.306047] Lustre: Skipped 2 previous similar messages [16697.565038] Lustre: server umount lustre-MDT0000 complete [16701.409132] LustreError: 506912:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16701.422549] LustreError: 506912:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [16701.515516] LustreError: 506100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857739 with bad export cookie 15390651837893849216 [16701.527053] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16701.538097] LustreError: 506100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16702.288258] Lustre: server umount lustre-MDT0001 complete [16715.808896] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [16719.288494] Key type lgssc unregistered [16719.595453] LNet: 510036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16719.607886] LNetError: 510036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16719.625584] LNet: Removed LNI 192.168.201.113@tcp [16720.556272] Key type .llcrypt unregistered [16720.558342] Key type ._llcrypt unregistered [16742.791785] Key type ._llcrypt registered [16742.797842] Key type .llcrypt registered [16742.956990] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [16744.618623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16744.666118] alg: No test for adler32 (adler32-zlib) [16746.048893] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [16746.608544] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [16748.536726] Key type lgssc registered [16750.700834] Lustre: Echo OBD driver; http://www.lustre.org/ [16762.291596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16762.358911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16764.340298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16771.032410] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16774.644332] LustreError: 511308:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16774.673664] LustreError: 511308:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [16779.757927] LustreError: 511307:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16784.391741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16784.870190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [16785.156871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16790.833472] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16798.167122] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16804.554476] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16811.357589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16811.722491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16818.138764] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16825.484939] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16845.957744] loop0: detected capacity change from 0 to 204800000 [16846.210810] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16846.222285] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16846.231773] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16851.853969] loop0: detected capacity change from 0 to 204800000 [16852.086650] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16852.094967] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16852.104666] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16857.181616] loop0: detected capacity change from 0 to 204800000 [16862.583735] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16862.755942] Lustre: fs15246-OST0000: new disk, initializing [16862.763782] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16862.775485] Lustre: 514924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16862.848024] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16870.453869] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16885.133791] loop1: detected capacity change from 0 to 204800000 [16892.945207] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16893.148917] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16893.247321] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16893.371823] Lustre: fs15246-MDT0000: new disk, initializing [16893.620437] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16893.682952] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16893.914323] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16894.946974] Lustre: 516050:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f79c7d43 x1874704763768064/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787857948 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16899.250368] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [16900.109080] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16900.125041] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16900.185740] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16924.158245] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16924.191589] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16929.134162] Lustre: server umount fs15246-MDT0000 complete [16929.262414] LustreError: 513433:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16934.385882] LustreError: 511303:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16939.497823] LustreError: 511303:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16939.905894] Lustre: server umount fs15246-OST0000 complete [16953.315117] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16953.323383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16953.345888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16954.864869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16958.991672] Lustre: server umount lustre-OST0000 complete [16964.064714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16964.072908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16964.090952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16964.095296] Lustre: Skipped 2 previous similar messages [16969.955425] Lustre: server umount lustre-MDT0000 complete [16970.218584] LustreError: 511307:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16970.245375] LustreError: 511307:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [16974.233667] LustreError: 511287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858012 with bad export cookie 3607573977379188519 [16974.237879] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16974.253361] LustreError: 511287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16974.717496] Lustre: server umount lustre-MDT0001 complete [16985.941177] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [16988.877300] Key type lgssc unregistered [16989.267479] LNet: 519149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16989.275304] LNetError: 519149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16989.300907] LNet: Removed LNI 192.168.201.113@tcp [16990.252473] Key type .llcrypt unregistered [16990.254416] Key type ._llcrypt unregistered [17015.782729] Key type ._llcrypt registered [17015.784081] Key type .llcrypt registered [17015.891225] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [17017.471944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17017.802189] alg: No test for adler32 (adler32-zlib) [17018.934524] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [17019.237964] LNet: Added LNI 192.168.201.113@tcp [8/256/0/180] [17020.896178] Key type lgssc registered [17022.267991] Lustre: Echo OBD driver; http://www.lustre.org/ [17065.040422] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [17074.970234] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [17114.274549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17114.321491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17115.625613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17115.656275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17115.831965] Lustre: lustre-MDT0000: new disk, initializing [17115.980762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17115.999953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17120.882807] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [17133.978252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17134.127899] Lustre: 523617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17134.210377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17134.214575] Lustre: Skipped 1 previous similar message [17134.356986] Lustre: lustre-MDT0001: new disk, initializing [17134.536437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17134.631398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17134.646286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17141.167250] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [17153.228628] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17159.609940] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17167.612899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17167.916364] Lustre: lustre-OST0000: new disk, initializing [17167.924561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17167.933644] Lustre: 525589:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17168.067469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17173.589268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17173.615838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17173.712229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17175.855341] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_default_debug -1 all [17185.829438] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17190.769807] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17191.056671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17195.439800] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17195.704197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17199.607285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17199.609023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17199.650609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17199.673371] LustreError: Skipped 1 previous similar message [17203.805615] Lustre: server umount lustre-OST0000 complete [17209.833774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17209.859329] Lustre: Skipped 1 previous similar message [17209.873153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17209.880440] Lustre: Skipped 2 previous similar messages [17214.669557] Lustre: server umount lustre-MDT0000 complete [17218.286714] LustreError: 523611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858256 with bad export cookie 10459651211250713974 [17218.288177] LustreError: MGC192.168.201.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17218.295913] LustreError: 523611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17218.686380] Lustre: server umount lustre-MDT0001 complete [17246.542805] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing set_hostid [17258.143985] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing load_modules_local [17290.232450] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16945 sec ========= 15:18:46 (1787858326) [17291.857514] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:18:48 (1787858328) === [17294.972134] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:18:51 (1787858331) === [17335.319730] Lustre: DEBUG MARKER: oleg113-server.virtnet: executing unload_modules_local [17338.873373] Key type lgssc unregistered [17339.193789] LNet: 532237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17339.202408] LNetError: 532237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17339.243634] LNet: Removed LNI 192.168.201.113@tcp [17339.962782] Key type .llcrypt unregistered [17339.965658] Key type ._llcrypt unregistered