[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 471087711 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003250] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.008348] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010021] pid_max: default: 32768 minimum: 301 [ 0.011156] LSM: Security Framework initializing [ 0.012065] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.014087] *** VALIDATE selinux *** [ 0.021609] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027102] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] *** VALIDATE tmpfs *** [ 0.032364] *** VALIDATE proc *** [ 0.033175] *** VALIDATE cgroup *** [ 0.034005] *** VALIDATE cgroup2 *** [ 0.035166] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036135] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038022] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.041267] debug: unmapping init [mem 0xffffffffa6059000-0xffffffffa6060fff] [ 0.043172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044789] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045027] ... version: 2 [ 0.045746] ... bit width: 48 [ 0.046016] ... generic registers: 4 [ 0.046783] ... value mask: 0000ffffffffffff [ 0.047014] ... max period: 00007fffffffffff [ 0.048013] ... fixed-purpose events: 3 [ 0.049008] ... event mask: 000000070000000f [ 0.050296] rcu: Hierarchical SRCU implementation. [ 0.052492] smp: Bringing up secondary CPUs ... [ 0.053690] x86: Booting SMP configuration: [ 0.054023] .... node #0, CPUs: #1 #2 #3 [ 0.057139] smp: Brought up 1 node, 4 CPUs [ 0.059010] smpboot: Max logical packages: 1 [ 0.060013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130435] node 0 deferred pages initialised in 68ms [ 0.134175] devtmpfs: initialized [ 0.135416] x86/mm: Memory block size: 128MB [ 0.138720] gcov: version magic: 0x41383552 [ 0.141339] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143285] pinctrl core: initialized pinctrl subsystem [ 0.144204] [ 0.144804] ************************************************************* [ 0.146016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148012] ** ** [ 0.150013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151008] ** ** [ 0.153007] ** This means that this kernel is built to expose internal ** [ 0.154007] ** IOMMU data structures, which may compromise security on ** [ 0.156011] ** your system. ** [ 0.159015] ** ** [ 0.161011] ** If you see this message and you are not debugging the ** [ 0.163016] ** kernel, report this immediately to your vendor! ** [ 0.164008] ** ** [ 0.166013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168013] ************************************************************* [ 0.170650] NET: Registered protocol family 16 [ 0.172477] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.174070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179040] cpuidle: using governor menu [ 0.180412] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181452] PCI: Using configuration type 1 for base access [ 0.183161] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.191024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192139] cryptd: max_cpu_qlen set to 1000 [ 0.195027] ACPI: Added _OSI(Module Device) [ 0.196013] ACPI: Added _OSI(Processor Device) [ 0.198032] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205395] ACPI: Interpreter enabled [ 0.206056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206779] ACPI: Using IOAPIC for interrupt routing [ 0.208094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210271] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.216828] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.218050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.220019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.222073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.226011] acpiphp: Slot [2] registered [ 0.226873] acpiphp: Slot [5] registered [ 0.227066] acpiphp: Slot [6] registered [ 0.227909] acpiphp: Slot [7] registered [ 0.229103] acpiphp: Slot [8] registered [ 0.230104] acpiphp: Slot [9] registered [ 0.232114] acpiphp: Slot [10] registered [ 0.232931] acpiphp: Slot [3] registered [ 0.233072] acpiphp: Slot [4] registered [ 0.233929] acpiphp: Slot [11] registered [ 0.235066] acpiphp: Slot [12] registered [ 0.235878] acpiphp: Slot [13] registered [ 0.237059] acpiphp: Slot [14] registered [ 0.237846] acpiphp: Slot [15] registered [ 0.238054] acpiphp: Slot [16] registered [ 0.239023] acpiphp: Slot [17] registered [ 0.239834] acpiphp: Slot [18] registered [ 0.241147] acpiphp: Slot [19] registered [ 0.242121] acpiphp: Slot [20] registered [ 0.244096] acpiphp: Slot [21] registered [ 0.245111] acpiphp: Slot [22] registered [ 0.247121] acpiphp: Slot [23] registered [ 0.248094] acpiphp: Slot [24] registered [ 0.249125] acpiphp: Slot [25] registered [ 0.251096] acpiphp: Slot [26] registered [ 0.252085] acpiphp: Slot [27] registered [ 0.253078] acpiphp: Slot [28] registered [ 0.255095] acpiphp: Slot [29] registered [ 0.256091] acpiphp: Slot [30] registered [ 0.257120] acpiphp: Slot [31] registered [ 0.259100] PCI host bridge to bus 0000:00 [ 0.260020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.262024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.265028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.267020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.269025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.272035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.274210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.277077] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.280312] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.289751] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.294421] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.297020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.299019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.301021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.304066] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.306638] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.308039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.310844] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.316016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.329019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.334015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.338751] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.346017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.354029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.384017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.392216] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.398021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.405070] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.423020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.431000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.436017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.443014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.458028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.469905] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.478034] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.486019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.504014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.514713] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.521016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.527017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.547018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.559014] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.568019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.576017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.595022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.608146] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.610358] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.612401] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.615477] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.618252] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.623174] iommu: Default domain type: Passthrough [ 0.625505] SCSI subsystem initialized [ 0.627152] ACPI: bus type USB registered [ 0.628096] usbcore: registered new interface driver usbfs [ 0.630184] usbcore: registered new interface driver hub [ 0.632163] usbcore: registered new device driver usb [ 0.634195] pps_core: LinuxPPS API ver. 1 registered [ 0.636013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.639130] PTP clock support registered [ 0.641169] EDAC MC: Ver: 3.0.0 [ 0.643113] PCI: Using ACPI for IRQ routing [ 0.644951] NetLabel: Initializing [ 0.647015] NetLabel: domain hash size = 128 [ 0.648011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.650092] NetLabel: unlabeled traffic allowed by default [ 0.653126] vgaarb: loaded [ 0.654295] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.656016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.664000] clocksource: Switched to clocksource kvm-clock [ 0.775131] VFS: Disk quotas dquot_6.6.0 [ 0.776669] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.779495] *** VALIDATE ramfs *** [ 0.780811] *** VALIDATE hugetlbfs *** [ 0.783293] pnp: PnP ACPI init [ 0.785746] pnp: PnP ACPI: found 6 devices [ 0.808548] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.812044] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.814441] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.816848] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.819378] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.821968] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.824839] NET: Registered protocol family 2 [ 0.827285] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.832157] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.835420] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.839957] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.843053] TCP: Hash tables configured (established 65536 bind 65536) [ 0.845734] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.848789] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.851653] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.854635] NET: Registered protocol family 1 [ 0.857649] RPC: Registered named UNIX socket transport module. [ 0.859812] RPC: Registered udp transport module. [ 0.860975] RPC: Registered tcp transport module. [ 0.862139] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.864132] NET: Registered protocol family 44 [ 0.866034] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.868217] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.870208] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.872340] PCI: CLS 0 bytes, default 64 [ 0.873966] Unpacking initramfs... [ 2.199765] debug: unmapping init [mem 0xffff96be3cc54000-0xffff96be3ffbffff] [ 2.203719] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.206249] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.209160] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.655370] Initialise system trusted keyrings [ 2.656566] Key type blacklist registered [ 2.657976] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.664836] zbud: loaded [ 2.667076] *** VALIDATE nfs *** [ 2.667896] *** VALIDATE nfs4 *** [ 2.668978] pstore: using deflate compression [ 2.671899] Platform Keyring initialized [ 2.752786] NET: Registered protocol family 38 [ 2.754668] Key type asymmetric registered [ 2.755957] Asymmetric key parser 'x509' registered [ 2.757859] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.760665] io scheduler mq-deadline registered [ 2.762085] io scheduler kyber registered [ 2.763452] io scheduler bfq registered [ 2.764923] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.767386] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.770029] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.772306] ACPI: Power Button [PWRF] [ 2.777223] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.783500] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.794558] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.800486] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.812443] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.838552] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.865517] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.870703] Non-volatile memory driver v1.3 [ 2.872429] Linux agpgart interface v0.103 [ 2.904414] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.907156] vda: detected capacity change from 0 to 74698752 [ 2.921396] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.924215] vdb: detected capacity change from 0 to 1073741824 [ 2.935032] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.936660] vdc: detected capacity change from 0 to 2621440000 [ 2.950145] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.952774] vdd: detected capacity change from 0 to 2621440000 [ 2.966835] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.969419] vde: detected capacity change from 0 to 4294967296 [ 2.982806] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.984946] vdf: detected capacity change from 0 to 4294967296 [ 2.991573] libphy: Fixed MDIO Bus: probed [ 2.996375] usbcore: registered new interface driver usbserial_generic [ 2.998616] usbserial: USB Serial support registered for generic [ 3.000475] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.003920] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.005391] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.007548] mousedev: PS/2 mouse device common for all mice [ 3.010261] rtc_cmos 00:05: RTC can wake from S4 [ 3.013090] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.013158] rtc_cmos 00:05: registered as rtc0 [ 3.017723] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.018699] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.021313] intel_pstate: CPU model not supported [ 3.025497] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.029894] hid: raw HID events driver (C) Jiri Kosina [ 3.032178] usbcore: registered new interface driver usbhid [ 3.034756] usbhid: USB HID core driver [ 3.036346] drop_monitor: Initializing network drop monitor service [ 3.038750] Initializing XFRM netlink socket [ 3.040835] NET: Registered protocol family 10 [ 3.043764] Segment Routing with IPv6 [ 3.044969] NET: Registered protocol family 17 [ 3.046694] mpls_gso: MPLS GSO support [ 3.052481] RAS: Correctable Errors collector initialized. [ 3.054909] AVX version of gcm_enc/dec engaged. [ 3.056857] AES CTR mode by8 optimization enabled [ 3.118907] sched_clock: Marking stable (3118887655, 0)->(3962761878, -843874223) [ 3.121098] registered taskstats version 1 [ 3.122633] Loading compiled-in X.509 certificates [ 3.124211] zswap: loaded using pool lzo/zbud [ 3.148610] Key type big_key registered [ 3.157252] Key type encrypted registered [ 3.159227] ima: No TPM chip found, activating TPM-bypass! [ 3.161608] ima: Allocated hash algorithm: sha1 [ 3.163633] ima: No architecture policies found [ 3.165984] evm: Initialising EVM extended attributes: [ 3.168345] evm: security.selinux [ 3.169870] evm: security.ima [ 3.171237] evm: security.capability [ 3.172796] evm: HMAC attrs: 0x1 [ 3.175426] rtc_cmos 00:05: setting system clock to 2026-09-03 14:01:44 UTC (1788444104) [ 3.182164] debug: unmapping init [mem 0xffffffffa7003000-0xffffffffa71fffff] [ 3.184914] debug: unmapping init [mem 0xffffffffa5d82000-0xffffffffa6058fff] [ 3.195107] Write protecting the kernel read-only data: 28672k [ 3.199347] debug: unmapping init [mem 0xffffffffa4403000-0xffffffffa45fffff] [ 3.202404] debug: unmapping init [mem 0xffffffffa4d14000-0xffffffffa4dfffff] [ 3.235912] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.245417] systemd[1]: Detected virtualization kvm. [ 3.247286] systemd[1]: Detected architecture x86-64. [ 3.249199] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.273554] systemd[1]: No hostname configured. [ 3.275086] systemd[1]: Set hostname to . [ 3.276493] random: systemd: uninitialized urandom read (16 bytes read) [ 3.278319] systemd[1]: Initializing machine ID from random generator. [ 3.316948] random: ln: uninitialized urandom read (6 bytes read) [ 3.401458] random: systemd: uninitialized urandom read (16 bytes read) [ 3.404485] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.409347] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.414413] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.958887] device-mapper: uevent: version 1.0.3 [ 3.961384] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.628727] random: fast init done Starting dracut initqueue hook... [ 4.703368] virtio_net virtio0 ens2: renamed from eth0 [ 4.783240] scsi host0: ata_piix [ 4.806759] scsi host1: ata_piix [ 4.807952] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.809735] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.122273] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.678994] random: crng init done [ 9.680401] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.992275] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.096068] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.274756] SELinux: Disabled at runtime. [ 11.329662] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.335173] systemd[1]: Detected virtualization kvm. [ 11.336524] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.779625] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.785085] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.789741] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.795352] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.798827] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.806265] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.812666] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 11.915396] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.264188] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.631836] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.635200] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.787141] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.797746] EDAC sbridge: Ver: 1.1.2 [ 14.171593] Key type dns_resolver registered [ 14.451558] NFS: Registering the id_resolver key type [ 14.453501] Key type id_resolver registered [ 14.454983] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg115-server login: [ 41.501388] libcfs: loading out-of-tree module taints kernel. [ 41.521892] Key type ._llcrypt registered [ 41.523549] Key type .llcrypt registered [ 41.581928] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 50.431322] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 51.280824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.289820] alg: No test for adler32 (adler32-zlib) [ 52.378070] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 52.741237] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 54.392247] Key type lgssc registered [ 55.176695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.601482] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 87.095322] hrtimer: interrupt took 6369057 ns [ 121.388067] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 137.032287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 137.053200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.710848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.791053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 138.963384] Lustre: lustre-MDT0000: new disk, initializing [ 139.091753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 139.107731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.051242] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.251975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.328433] Lustre: 6484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 160.351975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 160.354722] Lustre: Skipped 1 previous similar message [ 160.420948] Lustre: lustre-MDT0001: new disk, initializing [ 160.466796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 160.482938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 160.494379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.884617] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.525551] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.881211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 181.150290] Lustre: lustre-OST0000: new disk, initializing [ 181.154297] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 181.159705] Lustre: 8425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 181.229463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.915016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.933342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.044728] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 188.684026] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.386421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.582887] Lustre: lustre-OST0001: new disk, initializing [ 209.590575] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.606409] Lustre: 9497:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.770676] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.112609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.131487] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.224857] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 218.006829] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.945547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.939716] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 248.954837] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.510391] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing yml_node [ 259.617214] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 262.374648] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 265.355390] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 267.356955] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 10:06:07 EDT 2026 [ 287.127537] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 288.823992] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 309.217749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 309.219669] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.222721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.264516] Lustre: Skipped 3 previous similar messages [ 312.246662] Lustre: server umount lustre-MDT0000 complete [ 316.166406] LustreError: 7435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444417 with bad export cookie 3692618919612783589 [ 316.180267] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 316.182971] LustreError: 7435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 319.456735] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.458649] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.468195] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.468635] Lustre: Skipped 3 previous similar messages [ 319.486345] LustreError: 6492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 319.533767] Lustre: Skipped 1 previous similar message [ 321.504967] LustreError: 6493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.506646] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.527127] Lustre: Skipped 1 previous similar message [ 323.001859] Lustre: server umount lustre-MDT0001 complete [ 327.499419] Lustre: server umount lustre-OST0000 complete [ 331.476428] Lustre: server umount lustre-OST0001 complete [ 340.519618] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 352.590824] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 390.306091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.572317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 390.605595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 390.715608] Lustre: lustre-MDT0000: new disk, initializing [ 390.854237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.875849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 396.383390] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 407.498226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.601285] Lustre: 15054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 407.651616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 407.655090] Lustre: Skipped 1 previous similar message [ 407.722403] Lustre: lustre-MDT0001: new disk, initializing [ 407.833923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 407.885891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 407.903030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 412.881909] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 422.675494] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 428.940757] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 436.576281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 436.855722] Lustre: lustre-OST0000: new disk, initializing [ 436.867967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 436.888835] Lustre: 17021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 437.017787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.045991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 438.070764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 438.244472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 444.280952] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 455.170193] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 460.056580] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 460.400428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 465.122427] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 465.427287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 468.969612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 468.975453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.981604] LustreError: Skipped 1 previous similar message [ 468.985570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 469.001279] Lustre: Skipped 1 previous similar message [ 473.747140] Lustre: server umount lustre-OST0000 complete [ 479.208292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.213563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 479.217385] Lustre: Skipped 1 previous similar message [ 479.232019] Lustre: Skipped 2 previous similar messages [ 484.554307] Lustre: server umount lustre-MDT0000 complete [ 488.505274] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444589 with bad export cookie 3692618919612785122 [ 488.512930] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.524219] LustreError: 15045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 489.093507] Lustre: server umount lustre-MDT0001 complete [ 493.969928] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:09:53 (1788444593) [ 496.510358] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 498.569772] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:09:58 (1788444598) [ 506.353541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.924092] LustreError: 19433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.951728] LustreError: 19433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 507.089631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 511.409396] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 512.499531] LustreError: 19434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.531165] LustreError: 19434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 517.618613] LustreError: 19433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.502523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.727308] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 537.949282] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 544.630723] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 550.979560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 557.447740] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 564.624324] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 575.909680] Lustre: server umount lustre-OST0000 complete [ 585.193383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 585.196530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.230690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 585.240946] Lustre: Skipped 3 previous similar messages [ 591.496462] Lustre: server umount lustre-MDT0000 complete [ 595.430127] LustreError: 19433:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.455091] LustreError: 19433:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 596.309707] LustreError: 19414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444697 with bad export cookie 3692618919612786004 [ 596.325841] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.335861] LustreError: 19414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 606.852378] Lustre: server umount lustre-MDT0001 complete [ 622.746142] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 626.669654] Key type lgssc unregistered [ 627.154972] LNet: 22971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.159388] LNetError: 22971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.171905] LNet: Removed LNI 192.168.201.115@tcp [ 628.218140] Key type .llcrypt unregistered [ 628.221631] Key type ._llcrypt unregistered [ 657.513964] Key type ._llcrypt registered [ 657.517396] Key type .llcrypt registered [ 657.617707] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 658.990123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.125737] alg: No test for adler32 (adler32-zlib) [ 660.385750] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 660.609053] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 662.280193] Key type lgssc registered [ 663.489551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 672.466388] loop: module loaded [ 672.547218] loop0: detected capacity change from 0 to 204800000 [ 672.828458] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 672.858736] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 673.269938] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.990615] loop0: detected capacity change from 0 to 204800000 [ 679.162985] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 679.170980] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 679.180062] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.846866] loop0: detected capacity change from 0 to 204800000 [ 692.018155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 692.064731] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.352504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 693.446778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 693.631484] Lustre: lustre-MDT0000: new disk, initializing [ 693.728558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 693.747441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 699.325838] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 708.431567] loop1: detected capacity change from 0 to 204800000 [ 713.731123] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 714.018385] Lustre: lustre-OST07c6: new disk, initializing [ 714.022743] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 714.027301] Lustre: Skipped 1 previous similar message [ 714.032550] Lustre: 26166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 714.124077] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 720.049618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 720.070186] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 720.116459] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 720.172717] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 725.724102] Lustre: 26174:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 725.731673] Lustre: 26174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 725.736174] Lustre: 26174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 725.740976] Lustre: 26174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 725.746111] Lustre: 26174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 725.750800] Lustre: 26174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 743.151754] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 743.156602] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 743.159238] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 743.162973] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 743.166754] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 743.170337] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.899379] Lustre: 25153:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 747.912087] Lustre: 25153:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 747.921041] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 747.931088] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.942566] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 747.949224] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.968042] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 747.981057] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.992772] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 748.008293] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 748.026576] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 748.049108] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.311472] Lustre: 26174:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 751.324197] Lustre: 26174:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 751.331600] Lustre: 26174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 751.342099] Lustre: 26174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.355290] Lustre: 26174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 751.366345] Lustre: 26174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.374614] Lustre: 26174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 751.393712] Lustre: 26174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.407904] Lustre: 26174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 751.423422] Lustre: 26174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.437335] Lustre: 26174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 751.454718] Lustre: 26174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.875001] Lustre: 25153:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 756.889262] Lustre: 25153:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 756.904689] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 756.915340] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.920607] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 756.933799] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.942619] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 756.952697] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.959288] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 756.971486] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 756.982672] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 756.994245] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.737761] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 766.743100] Lustre: 25154:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 766.747897] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 766.752724] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 766.757134] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 766.760975] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 766.764202] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 766.767601] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 766.771025] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 766.774527] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 766.777610] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 766.780394] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 817.636294] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 817.658484] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.685603] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 822.762942] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 823.295127] Lustre: server umount lustre-OST07c6 complete [ 834.516822] Lustre: server umount lustre-MDT0000 complete [ 852.805486] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 856.680440] Key type lgssc unregistered [ 857.082781] LNet: 28727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.088543] LNetError: 28727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.110725] LNet: Removed LNI 192.168.201.115@tcp [ 857.913895] Key type .llcrypt unregistered [ 857.928689] Key type ._llcrypt unregistered [ 880.184440] Key type ._llcrypt registered [ 880.194493] Key type .llcrypt registered [ 880.364984] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 882.438301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.498472] alg: No test for adler32 (adler32-zlib) [ 883.593824] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 883.850968] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 885.545289] Key type lgssc registered [ 887.223372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.836196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.845395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.182737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.846397] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 905.699854] LustreError: 29999:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.717805] LustreError: 29999:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 910.818739] LustreError: 29998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 910.868925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.185900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 916.770415] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 925.133058] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 931.972950] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 939.200144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 939.646231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 941.682072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 947.700969] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 958.646714] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 972.769305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 972.771644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.786175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.334200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.341230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 978.898634] Lustre: server umount lustre-OST0000 complete [ 983.009934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 983.023546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.049744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.055763] Lustre: Skipped 1 previous similar message [ 985.570516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 985.572648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 988.909388] Lustre: server umount lustre-MDT0000 complete [ 990.689391] LustreError: 29994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.717540] LustreError: 29994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 993.021793] LustreError: 29979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445094 with bad export cookie 14775363444335189511 [ 993.030099] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 993.041837] LustreError: 29979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 993.758317] Lustre: server umount lustre-MDT0001 complete [ 1006.890868] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1009.860139] Key type lgssc unregistered [ 1010.200749] LNet: 33743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.210212] LNetError: 33743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.224802] LNet: Removed LNI 192.168.201.115@tcp [ 1011.049334] Key type .llcrypt unregistered [ 1011.052254] Key type ._llcrypt unregistered [ 1032.057101] Key type ._llcrypt registered [ 1032.058986] Key type .llcrypt registered [ 1032.175242] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1033.221844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.284556] alg: No test for adler32 (adler32-zlib) [ 1034.370487] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1034.562876] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1036.193430] Key type lgssc registered [ 1037.470776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1046.184302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1046.201907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.694760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.001087] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1057.252338] LustreError: 35013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1057.279660] LustreError: 35013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1060.611384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.983908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1065.813228] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1072.930445] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1078.802472] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1087.552296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1087.909327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.032647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1094.599949] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1101.731192] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1116.643356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1116.652908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.664954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1116.684190] Lustre: Skipped 1 previous similar message [ 1116.694812] Lustre: Skipped 1 previous similar message [ 1121.773669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1121.788299] Lustre: Skipped 1 previous similar message [ 1123.113209] Lustre: server umount lustre-OST0000 complete [ 1127.396438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1127.405792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.434961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1132.002212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.003859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1132.050555] Lustre: Skipped 1 previous similar message [ 1133.786933] Lustre: server umount lustre-MDT0000 complete [ 1137.122058] LustreError: 35032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1137.157862] LustreError: 35032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1138.250986] LustreError: 34995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445239 with bad export cookie 9244494442554779335 [ 1138.267065] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1138.599338] Lustre: server umount lustre-MDT0001 complete [ 1151.999961] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1155.114669] Key type lgssc unregistered [ 1155.495252] LNet: 38571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.508605] LNetError: 38571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.538355] LNet: Removed LNI 192.168.201.115@tcp [ 1156.641505] Key type .llcrypt unregistered [ 1156.645420] Key type ._llcrypt unregistered [ 1180.359976] Key type ._llcrypt registered [ 1180.363800] Key type .llcrypt registered [ 1180.483373] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1181.998192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1182.272173] alg: No test for adler32 (adler32-zlib) [ 1183.422628] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1183.749506] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1185.504493] Key type lgssc registered [ 1186.931444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1198.820418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1198.895207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.726706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.954132] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1209.828787] LustreError: 39997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1209.848186] LustreError: 39997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1214.946375] LustreError: 39996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.601295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.855346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1220.946609] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1229.643911] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1235.745869] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1242.746408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1243.072753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1245.114758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1249.763485] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1258.482232] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1271.308605] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 1271.789777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.809657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.817502] Lustre: Skipped 2 previous similar messages [ 1271.835206] Lustre: Skipped 2 previous similar messages [ 1275.841945] Lustre: server umount lustre-MDT0000 complete [ 1276.376070] LustreError: 39992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.179579] LustreError: 39975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445382 with bad export cookie 9039985058588683860 [ 1281.183966] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1281.195841] LustreError: 39975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1281.493730] LustreError: 39993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.511897] LustreError: 39993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1281.534770] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [ 1282.021222] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.049119] LustreError: 39991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.049800] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1286.075326] LustreError: 39991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1286.118752] Lustre: Skipped 1 previous similar message [ 1287.758922] Lustre: server umount lustre-MDT0001 complete [ 1303.640847] Lustre: server umount lustre-OST0000 complete [ 1323.344204] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1325.581474] Key type lgssc unregistered [ 1325.777458] LNet: 43859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.779990] LNetError: 43859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.795128] LNet: Removed LNI 192.168.201.115@tcp [ 1326.305434] Key type .llcrypt unregistered [ 1326.307116] Key type ._llcrypt unregistered [ 1352.638752] Key type ._llcrypt registered [ 1352.646531] Key type .llcrypt registered [ 1352.786926] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1354.284320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1354.538224] alg: No test for adler32 (adler32-zlib) [ 1355.837296] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1356.226239] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1357.976368] Key type lgssc registered [ 1359.813894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1370.721269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1370.752797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.272929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.010337] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1381.860312] LustreError: 45279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.892122] LustreError: 45279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1385.891731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.440285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1391.284633] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1399.422354] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1404.520210] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1411.087521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.533110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.492604] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1417.725227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1425.134451] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1436.131090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1436.143159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.160412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1436.187546] Lustre: Skipped 1 previous similar message [ 1436.204276] Lustre: Skipped 1 previous similar message [ 1437.198476] Lustre: lustre-OST0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 1440.467729] Lustre: server umount lustre-OST0000 complete [ 1461.730162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.731105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.748512] Lustre: Skipped 1 previous similar message [ 1465.082364] Lustre: server umount lustre-MDT0000 complete [ 1466.849251] LustreError: 47773:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.856178] LustreError: 47773:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1469.973258] LustreError: 45259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445571 with bad export cookie 10845381908810744238 [ 1469.975469] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1469.981042] LustreError: 45259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1476.247622] Lustre: server umount lustre-MDT0001 complete [ 1491.585780] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1496.294300] Key type lgssc unregistered [ 1496.773878] LNet: 48992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1496.785711] LNetError: 48992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.834065] LNet: Removed LNI 192.168.201.115@tcp [ 1499.515778] Key type .llcrypt unregistered [ 1499.517928] Key type ._llcrypt unregistered [ 1524.068980] Key type ._llcrypt registered [ 1524.072166] Key type .llcrypt registered [ 1524.207224] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1525.385860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1525.432534] alg: No test for adler32 (adler32-zlib) [ 1526.423249] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1526.672103] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1528.417048] Key type lgssc registered [ 1529.630503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1537.122902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1537.138956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.688979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.385452] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1548.266061] LustreError: 50403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.294705] LustreError: 50403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1552.335051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1552.664686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1557.035423] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1565.687481] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1570.683515] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1577.156333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1577.475589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1583.223926] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1583.602512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1590.717128] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1596.707569] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1601.435220] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1614.585070] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0310h28m33s [ 1616.161273] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1619.441319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1619.445557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.451613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1619.484953] Lustre: Skipped 1 previous similar message [ 1620.580155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1620.586061] Lustre: Skipped 4 previous similar messages [ 1623.052628] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 1624.988692] Lustre: server umount lustre-MDT0000 complete [ 1627.119637] LustreError: 50403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.953753] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1630.687440] LustreError: 53097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1630.727894] LustreError: 53097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1633.250341] LustreError: 53097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.274679] LustreError: 53097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1636.597257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1636.829799] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1637.134664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.198022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1638.220748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1641.789360] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1645.184724] Lustre: DEBUG MARKER: Wait for df (36739) ... [ 1646.792346] Lustre: DEBUG MARKER: done [ 1652.705285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1652.708461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.712632] Lustre: Skipped 1 previous similar message [ 1652.714761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1652.716775] Lustre: Skipped 1 previous similar message [ 1653.736246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.749158] Lustre: server umount lustre-OST0000 complete [ 1663.970752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.974266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1663.975821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1663.992716] Lustre: Skipped 1 previous similar message [ 1664.005833] Lustre: Skipped 3 previous similar messages [ 1666.863229] Lustre: server umount lustre-MDT0000 complete [ 1670.578770] LustreError: 53028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445771 with bad export cookie 12051059379721833871 [ 1670.584390] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.594130] LustreError: 53028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1670.930953] Lustre: server umount lustre-MDT0001 complete [ 1682.171427] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1684.985559] Key type lgssc unregistered [ 1685.245461] LNet: 55331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.260214] LNetError: 55331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.277935] LNet: Removed LNI 192.168.201.115@tcp [ 1686.341001] Key type .llcrypt unregistered [ 1686.343611] Key type ._llcrypt unregistered [ 1727.630363] Key type ._llcrypt registered [ 1727.632686] Key type .llcrypt registered [ 1727.738899] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1729.406658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1729.426355] alg: No test for adler32 (adler32-zlib) [ 1730.468896] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1730.746665] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1732.416191] Key type lgssc registered [ 1733.899657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1742.733754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1742.764750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.176315] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1744.220454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1744.418965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1749.619566] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1758.881327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.049771] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1759.085516] Lustre: 57540:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1759.357741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1764.324557] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1773.999608] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1780.297553] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1787.499369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1787.657845] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1787.852382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.738913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1794.522933] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 1802.911272] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1811.451371] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0310h31m50s [ 1813.035482] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1820.206527] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1822.955198] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1823.019880] LustreError: 59687:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1843.202132] Lustre: lustre-MDT0000: Client 1963c449-da5b-4b5f-be5a-60865b4c890f (at 192.168.201.15@tcp) reconnecting [ 1863.056131] LustreError: 59687:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1865.919899] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1868.570602] Lustre: DEBUG MARKER: done [ 1875.430777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1875.455893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.478939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1876.400423] Lustre: server umount lustre-OST0000 complete [ 1881.057428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1881.062038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1881.069570] Lustre: Skipped 2 previous similar messages [ 1881.091195] Lustre: Skipped 2 previous similar messages [ 1882.151450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.161427] Lustre: Skipped 1 previous similar message [ 1887.206949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.219740] Lustre: Skipped 1 previous similar message [ 1893.349773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.369555] Lustre: Skipped 1 previous similar message [ 1895.392175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1895.775356] Lustre: server umount lustre-MDT0000 complete [ 1900.565890] LustreError: 57532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446001 with bad export cookie 8212841913835249343 [ 1900.570939] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.581725] LustreError: 57532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1901.328833] Lustre: server umount lustre-MDT0001 complete [ 1916.392390] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 1920.088481] Key type lgssc unregistered [ 1920.461913] LNet: 61873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.471911] LNetError: 61873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.492289] LNet: Removed LNI 192.168.201.115@tcp [ 1921.798182] Key type .llcrypt unregistered [ 1921.800090] Key type ._llcrypt unregistered [ 1972.242668] Key type ._llcrypt registered [ 1972.245627] Key type .llcrypt registered [ 1972.385193] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 1974.158460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1974.188421] alg: No test for adler32 (adler32-zlib) [ 1975.294339] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1975.657502] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 1977.424299] Key type lgssc registered [ 1978.659866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1982.820485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1982.862874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.211294] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1984.242788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1984.492780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1985.364508] Lustre: Failing over lustre-MDT0000 [ 1985.811552] Lustre: server umount lustre-MDT0000 complete [ 1992.692840] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:34:52 (1788446092) [ 1998.988667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.261983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.289093] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2012.172578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.245163] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2012.277684] Lustre: 64672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2012.362460] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2012.493447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2016.778445] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2023.169066] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2028.200191] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2034.678798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2034.883949] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2035.116654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2040.518825] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2044.606432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2048.191502] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2055.373990] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2063.841323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2063.846167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.855140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2065.394302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2065.399824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2065.414562] Lustre: Skipped 1 previous similar message [ 2065.428576] Lustre: Skipped 2 previous similar messages [ 2068.356273] Lustre: server umount lustre-MDT0000 complete [ 2072.260974] LustreError: 64665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446173 with bad export cookie 3691871504668347528 [ 2072.267730] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2072.272689] LustreError: 64665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2072.518349] Lustre: server umount lustre-MDT0001 complete [ 2075.766816] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2080.344248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2088.695973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.977481] Lustre: 63598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446176/real 1788446176] req@0000000009809326 x1875321632628736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788446192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.025533] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.024172] Lustre: 63596:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446178/real 1788446178] req@0000000070a4656a x1875321632628864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788446194 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2096.096188] Lustre: 63596:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446181/real 1788446182] req@000000001366cb99 x1875321632628992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788446197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2097.214657] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x333c2992b824d56b [ 2097.230881] Lustre: MGC192.168.201.115@tcp: Connection restored to 0@lo (at 0@lo) [ 2097.686256] LustreError: 66525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.713466] LustreError: 66525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2097.809432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.954244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2098.784191] Lustre: 63598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446183/real 1788446183] req@00000000aeea6a38 x1875321632629248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788446199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2100.839703] LustreError: 66526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.863667] LustreError: 68447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.886706] LustreError: 68447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2102.466919] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2102.869362] Lustre: 63598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446188/real 1788446188] req@0000000008721764 x1875321632629376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788446204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2107.369093] LustreError: 68446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.398605] LustreError: 68446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2112.482360] LustreError: 68447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.499024] LustreError: 68447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2112.501506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.888190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2117.147743] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2118.129371] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2120.164903] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2124.388665] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2130.253575] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2136.713172] Lustre: server umount lustre-MDT0000 complete [ 2138.593709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2138.602251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.621442] LustreError: 66526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.638428] LustreError: 64665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446240 with bad export cookie 3691871504668349803 [ 2139.639623] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2139.650040] LustreError: 64665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2140.643388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2141.944704] Lustre: server umount lustre-MDT0001 complete [ 2149.192790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2156.948926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.153024] LustreError: 63594:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b96a60bb x1875321632656128/t0(0) o250->MGC192.168.201.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2165.508609] LustreError: 66525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.518963] LustreError: 66525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2165.581816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.630654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2169.837485] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2177.229231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.695808] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2182.036536] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2189.230264] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2194.386885] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2200.546626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.552751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.576618] Lustre: Skipped 4 previous similar messages [ 2200.596053] Lustre: Skipped 1 previous similar message [ 2202.168716] Lustre: server umount lustre-MDT0000 complete [ 2205.672291] LustreError: 71125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.686159] LustreError: 71125:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2205.947423] LustreError: 64667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446307 with bad export cookie 3691871504668351182 [ 2205.950972] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.961502] LustreError: 64667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2206.450957] Lustre: server umount lustre-MDT0001 complete [ 2212.594595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2215.706460] Lustre: DEBUG MARKER: files compared the same [ 2227.752819] Lustre: server umount lustre-OST0000 complete [ 2244.626787] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2247.619463] Key type lgssc unregistered [ 2248.028737] LNet: 74405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.031899] LNetError: 74405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.058180] LNet: Removed LNI 192.168.201.115@tcp [ 2249.080805] Key type .llcrypt unregistered [ 2249.083462] Key type ._llcrypt unregistered [ 2273.534641] Key type ._llcrypt registered [ 2273.536835] Key type .llcrypt registered [ 2273.663626] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2275.060288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2275.125584] alg: No test for adler32 (adler32-zlib) [ 2276.140978] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2276.418914] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2278.056244] Key type lgssc registered [ 2279.528162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2286.771897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2286.811346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.081493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.886781] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug +malloc all [ 2298.343611] LustreError: 75823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.361403] LustreError: 75823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2299.478335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2299.699702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2303.778872] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug +malloc all [ 2311.524979] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2317.908341] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2324.676142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2325.101114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.835297] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug +malloc all [ 2333.288100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2337.855824] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2349.026477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2349.033443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.047811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.051666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.052808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.087847] Lustre: Skipped 1 previous similar message [ 2350.860724] Lustre: server umount lustre-OST0000 complete [ 2355.168741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.180220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.183065] Lustre: Skipped 1 previous similar message [ 2355.194618] Lustre: Skipped 1 previous similar message [ 2360.290202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2360.303153] Lustre: Skipped 1 previous similar message [ 2360.676629] Lustre: server umount lustre-MDT0000 complete [ 2364.032903] LustreError: 75803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446465 with bad export cookie 5074141279264099003 [ 2364.043048] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2364.047568] LustreError: 75803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2364.460250] Lustre: server umount lustre-MDT0001 complete [ 2373.804782] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2375.992705] Key type lgssc unregistered [ 2376.238853] LNet: 79358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2376.253892] LNetError: 79358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2376.269264] LNet: Removed LNI 192.168.201.115@tcp [ 2377.056211] Key type .llcrypt unregistered [ 2377.057760] Key type ._llcrypt unregistered [ 2399.353658] Key type ._llcrypt registered [ 2399.359690] Key type .llcrypt registered [ 2399.461733] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2400.737311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2400.775352] alg: No test for adler32 (adler32-zlib) [ 2401.848250] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2402.101080] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2403.800287] Key type lgssc registered [ 2405.019562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2412.907732] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2412.932525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2428.512864] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2433.632383] LustreError: 80750:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [ 2460.321196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.533703] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2474.222125] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2481.249623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2481.339876] Lustre: *** cfs_fail_loc=706, val=0*** [ 2481.654501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.709719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2486.498443] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2486.763189] LustreError: 80776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.781108] LustreError: 80776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2491.881473] LustreError: 80775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.955219] LustreError: 81834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.607931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.043154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2499.936706] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2506.983942] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2512.251775] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2516.452266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.465707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2516.472729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2516.489957] Lustre: Skipped 1 previous similar message [ 2520.490247] Lustre: server umount lustre-OST0000 complete [ 2526.699063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2526.718365] Lustre: Skipped 2 previous similar messages [ 2526.721684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2526.722025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2526.763638] Lustre: Skipped 1 previous similar message [ 2530.700251] Lustre: server umount lustre-MDT0000 complete [ 2531.815821] LustreError: 81839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.832280] LustreError: 81839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2534.035059] LustreError: 82575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446635 with bad export cookie 12242609751672945331 [ 2534.039246] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2534.045548] LustreError: 82575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2534.336499] Lustre: server umount lustre-MDT0001 complete [ 2545.968364] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2548.497611] Key type lgssc unregistered [ 2548.759954] LNet: 84368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.764243] LNetError: 84368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.782782] LNet: Removed LNI 192.168.201.115@tcp [ 2549.668996] Key type .llcrypt unregistered [ 2549.671632] Key type ._llcrypt unregistered [ 2577.503863] Key type ._llcrypt registered [ 2577.506786] Key type .llcrypt registered [ 2577.598427] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2578.979190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2579.015709] alg: No test for adler32 (adler32-zlib) [ 2580.362353] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2580.676175] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2582.312599] Key type lgssc registered [ 2583.518178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2591.985536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2592.004935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.317401] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2605.527922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.216774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2610.629767] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2611.685976] LustreError: 86470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.707764] LustreError: 86470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2615.785499] LustreError: 86474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2615.807694] LustreError: 86474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2619.486800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2619.771289] LustreError: 86473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2619.789649] LustreError: 86473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2619.799899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.994670] LustreError: 86469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2625.031318] LustreError: 86469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2625.942099] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2630.117248] LustreError: 86474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.142452] LustreError: 86474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2634.358337] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2634.370332] Lustre: MGS: Client 8c417085-ced8-4590-a249-c4f875758e3c (at 0@lo) reconnecting [ 2634.399416] LustreError: 86484:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2634.590363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.668922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2639.167852] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2655.204082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.208558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2655.226792] Lustre: Skipped 1 previous similar message [ 2655.241103] Lustre: Skipped 1 previous similar message [ 2658.287494] Lustre: server umount lustre-OST0000 complete [ 2665.441367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.456644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2665.458934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.561926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.568418] Lustre: Skipped 2 previous similar messages [ 2675.682444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.688628] Lustre: Skipped 1 previous similar message [ 2676.192591] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2676.332490] Lustre: server umount lustre-MDT0000 complete [ 2679.972870] Lustre: server umount lustre-MDT0001 complete [ 2682.864240] Lustre: server umount MGS complete [ 2694.615312] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:46:34 (1788446794) [ 2722.151722] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 2731.168610] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2763.483568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.532033] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2777.199123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.370386] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2777.402510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2777.463814] Lustre: lustre-MDT0001: new disk, initializing [ 2777.530533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2777.556211] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2777.567416] Lustre: 93639:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2777.575332] LustreError: 93639:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2780.964860] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2789.301168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2789.525907] Lustre: lustre-OST0000: new disk, initializing [ 2789.529527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2789.534921] Lustre: 94404:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2789.613727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.546771] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2797.665834] Lustre: 94805:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000036669a0 x1875322267017728/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788446914 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2805.140819] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2811.017514] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.026157] Lustre: MGS: Client fe4713d2-776e-4718-9ce2-eb3c3789c323 (at 0@lo) reconnecting [ 2811.068555] Lustre: 92906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2811.088930] LustreError: 93625:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2811.128796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2811.263779] Lustre: lustre-MDT0000: new disk, initializing [ 2811.575494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.623922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2811.661431] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2812.720479] Lustre: 95472:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000000a7cdd45 x1875322267024384/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788446930 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2815.430208] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2817.771527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2817.787547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2817.867890] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2839.522131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2839.527988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.540901] Lustre: Skipped 1 previous similar message [ 2839.554614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2843.110204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.228836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2848.239252] Lustre: Skipped 3 previous similar messages [ 2852.832128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2853.036719] Lustre: server umount lustre-OST0000 complete [ 2858.467754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.470290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2863.609995] Lustre: server umount lustre-MDT0000 complete [ 2873.959368] Lustre: server umount lustre-MDT0001 complete [ 2877.162184] Lustre: server umount MGS complete [ 2886.099413] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:49:46 (1788446986) [ 2901.075353] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 2903.144281] Key type lgssc unregistered [ 2903.388937] LNet: 97973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2903.394650] LNetError: 97973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2903.408718] LNet: Removed LNI 192.168.201.115@tcp [ 2904.131681] Key type .llcrypt unregistered [ 2904.134032] Key type ._llcrypt unregistered [ 2919.143365] Key type ._llcrypt registered [ 2919.145274] Key type .llcrypt registered [ 2919.239569] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 2919.966117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2920.020681] alg: No test for adler32 (adler32-zlib) [ 2921.003253] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2921.171753] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 2922.848305] Key type lgssc registered [ 2923.764811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2930.041748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2930.060868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.516654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2935.449138] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2940.901286] LustreError: 99177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.921108] LustreError: 99177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2942.996418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.357017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2947.336876] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2954.270062] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2959.877432] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2967.077148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2967.318423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2972.947982] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 2976.569266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2980.101379] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2990.050548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2990.067430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2990.090325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2992.100663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2992.128030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.166061] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2994.140556] Lustre: server umount lustre-OST0000 complete [ 2999.778938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2999.786681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.798341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2999.801047] Lustre: Skipped 1 previous similar message [ 3002.344069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.344717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3005.004914] Lustre: server umount lustre-MDT0000 complete [ 3007.460327] LustreError: 99172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3007.483148] LustreError: 99172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3008.214540] LustreError: 99158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447109 with bad export cookie 11920337557386176965 [ 3008.234552] LustreError: 99158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3008.248391] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3008.524906] Lustre: server umount lustre-MDT0001 complete [ 3019.798103] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3022.721844] Key type lgssc unregistered [ 3022.952829] LNet: 102711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3022.960530] LNetError: 102711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3022.969849] LNet: Removed LNI 192.168.201.115@tcp [ 3023.853548] Key type .llcrypt unregistered [ 3023.862168] Key type ._llcrypt unregistered [ 3040.425787] Key type ._llcrypt registered [ 3040.429115] Key type .llcrypt registered [ 3040.527685] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3041.832497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3041.875387] alg: No test for adler32 (adler32-zlib) [ 3042.909648] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3043.071664] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3044.776166] Key type lgssc registered [ 3046.514917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3054.129442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3054.138452] LustreError: 103787:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3054.286517] LustreError: 103800:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3054.293161] LustreError: 103787:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4864 [ 3054.304329] LustreError: 103787:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3054.320451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.320708] LustreError: 103787:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3055.866979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3065.055053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.331306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.673985] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3076.550408] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing lsmod [ 3079.984487] LustreError: 105007:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3079.993172] LustreError: 105008:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3080.013877] LustreError: 105007:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3080.027727] LustreError: 105007:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3080.032936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3080.044024] LustreError: 105007:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3080.472141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.545436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3085.794655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3085.806335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3085.833180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3086.818391] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.818851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3086.839235] Lustre: Skipped 1 previous similar message [ 3086.847172] Lustre: Skipped 1 previous similar message [ 3089.271343] Lustre: server umount lustre-MDT0000 complete [ 3089.964333] LustreError: 103838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3089.995366] LustreError: 103838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3092.481221] LustreError: 105193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447193 with bad export cookie 8897891113639854281 [ 3092.482823] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.490518] LustreError: 105193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3092.751958] Lustre: server umount lustre-MDT0001 complete [ 3106.619623] Lustre: server umount lustre-OST0000 complete [ 3114.217555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.616452] LustreError: 106072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.742143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.918571] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3120.113165] LustreError: 106073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.214699] LustreError: 106072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3127.998155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3128.249775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3132.380937] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3139.236620] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3144.164547] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3150.108452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3150.353612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3157.934989] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3159.602691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3166.664645] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3174.883477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3174.887520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3174.890058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3174.918890] Lustre: Skipped 1 previous similar message [ 3180.011426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3180.028761] Lustre: Skipped 2 previous similar messages [ 3180.510414] Lustre: server umount lustre-OST0000 complete [ 3188.194954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.198672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.212145] Lustre: Skipped 1 previous similar message [ 3188.231427] Lustre: Skipped 1 previous similar message [ 3191.383629] Lustre: server umount lustre-MDT0000 complete [ 3193.314195] LustreError: 106073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.330901] LustreError: 106073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3195.775492] LustreError: 106053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447297 with bad export cookie 8897891113639855212 [ 3195.779877] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3195.793458] LustreError: 106053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3196.042202] Lustre: server umount lustre-MDT0001 complete [ 3206.323354] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3209.104694] Key type lgssc unregistered [ 3209.428111] LNet: 109608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3209.433674] LNetError: 109608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3209.448855] LNet: Removed LNI 192.168.201.115@tcp [ 3210.082785] Key type .llcrypt unregistered [ 3210.084590] Key type ._llcrypt unregistered [ 3231.754389] Key type ._llcrypt registered [ 3231.756836] Key type .llcrypt registered [ 3231.857133] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3233.034148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3233.043392] alg: No test for adler32 (adler32-zlib) [ 3233.960807] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3234.150655] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3235.824330] Key type lgssc registered [ 3236.712862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3244.336881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3244.364746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.061636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3251.711831] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3256.291705] LustreError: 111018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3256.318923] LustreError: 111018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3261.417604] LustreError: 111017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.022862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.334640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3266.046746] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3272.021938] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3278.218598] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3284.843712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.170986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3290.184479] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3291.330158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3297.249446] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3301.317422] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3304.761594] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3312.103080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3312.104613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.112605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3312.155801] Lustre: Skipped 1 previous similar message [ 3317.231346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3317.238738] Lustre: Skipped 2 previous similar messages [ 3322.347898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3322.354280] Lustre: Skipped 1 previous similar message [ 3323.367581] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3323.479450] Lustre: server umount lustre-OST0000 complete [ 3327.457750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3327.462021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.467619] Lustre: Skipped 1 previous similar message [ 3327.481033] Lustre: Skipped 1 previous similar message [ 3332.577621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3332.586037] Lustre: Skipped 1 previous similar message [ 3332.884953] Lustre: server umount lustre-MDT0000 complete [ 3336.622240] LustreError: 110999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447437 with bad export cookie 1228031285011894490 [ 3336.626958] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3336.638688] LustreError: 110999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3337.067454] Lustre: server umount lustre-MDT0001 complete [ 3347.709174] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3350.859760] Key type lgssc unregistered [ 3351.125820] LNet: 114661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3351.133817] LNetError: 114661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3352.181049] LNet: Removed LNI 192.168.201.115@tcp [ 3352.982414] Key type .llcrypt unregistered [ 3352.984313] Key type ._llcrypt unregistered [ 3372.367984] Key type ._llcrypt registered [ 3372.370310] Key type .llcrypt registered [ 3372.449218] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3373.654749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3373.712766] alg: No test for adler32 (adler32-zlib) [ 3374.874477] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3375.277244] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3377.009249] Key type lgssc registered [ 3377.942496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3386.944166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3386.980427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3388.478283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.789463] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3398.633100] LustreError: 115885:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3398.660432] LustreError: 115885:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3400.981248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.331540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3404.733056] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3411.785902] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3417.428825] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3425.494566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3425.739974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.007304] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3433.970799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3441.042579] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3449.315943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.318467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3449.326843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3449.333410] Lustre: Skipped 1 previous similar message [ 3450.800168] Lustre: server umount lustre-OST0000 complete [ 3454.436333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.449796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.450269] Lustre: Skipped 1 previous similar message [ 3454.455981] Lustre: Skipped 1 previous similar message [ 3459.554894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3459.572665] Lustre: Skipped 2 previous similar messages [ 3460.910833] Lustre: server umount lustre-MDT0000 complete [ 3464.359593] LustreError: 115866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447565 with bad export cookie 11964389597688980064 [ 3464.374229] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3464.379946] LustreError: 115866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3464.721698] Lustre: server umount lustre-MDT0001 complete [ 3477.197525] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3480.303378] Key type lgssc unregistered [ 3480.579794] LNet: 119418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3480.591503] LNetError: 119418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3480.617850] LNet: Removed LNI 192.168.201.115@tcp [ 3481.503070] Key type .llcrypt unregistered [ 3481.504379] Key type ._llcrypt unregistered [ 3508.493869] Key type ._llcrypt registered [ 3508.506448] Key type .llcrypt registered [ 3508.636577] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3509.683959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3509.767140] alg: No test for adler32 (adler32-zlib) [ 3510.951315] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3511.210204] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3512.920198] Key type lgssc registered [ 3514.368550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3522.576869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3522.602469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.979428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.758194] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3534.324205] LustreError: 120835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3534.352210] LustreError: 120835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3538.318596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.054312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3543.780318] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3550.139957] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3555.612148] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3562.887921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3563.232761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.402341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3569.562416] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3577.138921] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3585.560920] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3589.216353] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3589.220889] Lustre: Skipped 2 previous similar messages [ 3591.787574] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3591.818554] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3592.835427] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3592.837914] Lustre: Skipped 2 previous similar messages [ 3598.432254] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3598.437790] Lustre: Skipped 1 previous similar message [ 3603.626716] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3603.646111] Lustre: Skipped 2 previous similar messages [ 3607.286383] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.15@tcp 192.168.0.[2,10]@tcp [ 3611.935538] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.15/24@tcp in log lustre-MDT0000 [ 3611.943671] Lustre: Skipped 2 previous similar messages [ 3616.171561] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.15/24@tcp [ 3616.185152] Lustre: Skipped 2 previous similar messages [ 3620.834821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3620.840686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3620.844835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3620.857616] Lustre: Skipped 1 previous similar message [ 3620.875743] Lustre: Skipped 1 previous similar message [ 3623.906502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3623.915062] Lustre: Skipped 1 previous similar message [ 3626.874177] Lustre: server umount lustre-OST0000 complete [ 3631.075152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3631.085089] LustreError: Skipped 1 previous similar message [ 3631.092111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3631.104887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.146048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.147777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3636.830763] Lustre: server umount lustre-MDT0000 complete [ 3639.265226] LustreError: 120829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.318184] LustreError: 120829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3640.147835] LustreError: 123881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447741 with bad export cookie 1887983765470190033 [ 3640.158038] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3640.512731] Lustre: server umount lustre-MDT0001 complete [ 3649.185587] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3651.519992] Key type lgssc unregistered [ 3651.744834] LNet: 125748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3651.753232] LNetError: 125748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3651.768136] LNet: Removed LNI 192.168.201.115@tcp [ 3652.395168] Key type .llcrypt unregistered [ 3652.397383] Key type ._llcrypt unregistered [ 3676.186948] Key type ._llcrypt registered [ 3676.190882] Key type .llcrypt registered [ 3676.338755] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3677.607620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3677.719056] alg: No test for adler32 (adler32-zlib) [ 3678.747730] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3678.969058] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3680.624629] Key type lgssc registered [ 3681.833809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3689.771714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3689.797080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.276995] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3691.304363] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.15/24@tcp [ 3691.314676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3695.569534] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3700.708579] LustreError: 127287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3700.719651] LustreError: 127287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3703.675696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.018312] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3704.040843] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.15/24@tcp [ 3704.089587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3708.861048] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3716.351888] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3722.956556] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3728.765444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3729.166392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.822308] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3736.405323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3744.052147] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3758.052814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3758.066503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.077475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3760.103732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3760.107244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3760.107354] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3763.475685] Lustre: server umount lustre-OST0000 complete [ 3770.339587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3770.346743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3770.362193] Lustre: Skipped 1 previous similar message [ 3770.369852] Lustre: Skipped 1 previous similar message [ 3773.521988] Lustre: server umount lustre-MDT0000 complete [ 3775.467237] LustreError: 128536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3775.490178] LustreError: 128536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3776.978479] LustreError: 127267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447878 with bad export cookie 11586719959828027911 [ 3776.997687] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3777.461253] Lustre: server umount lustre-MDT0001 complete [ 3788.847897] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 3791.282164] Key type lgssc unregistered [ 3791.541897] LNet: 130879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3791.548764] LNetError: 130879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3791.570549] LNet: Removed LNI 192.168.201.115@tcp [ 3792.363727] Key type .llcrypt unregistered [ 3792.365454] Key type ._llcrypt unregistered [ 3817.772561] Key type ._llcrypt registered [ 3817.775685] Key type .llcrypt registered [ 3817.904688] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 3835.859109] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 3837.367380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3837.441591] alg: No test for adler32 (adler32-zlib) [ 3838.692846] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3839.103751] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 3840.984214] Key type lgssc registered [ 3842.753317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3878.273214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3878.333209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.702846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3879.762648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3879.863815] Lustre: lustre-MDT0000: new disk, initializing [ 3879.926412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.946129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3883.910606] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3892.819408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.940034] Lustre: 134280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3893.004460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3893.008350] Lustre: Skipped 1 previous similar message [ 3893.063405] Lustre: lustre-MDT0001: new disk, initializing [ 3893.201864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3893.254722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3893.269833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3897.839621] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3907.986261] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3915.136897] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3921.965196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3922.238429] Lustre: lustre-OST0000: new disk, initializing [ 3922.242171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3922.250299] Lustre: 136249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3922.349712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3927.504720] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3929.685078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3929.698057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3929.794432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3936.578866] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3941.888396] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3942.163677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3946.518533] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3946.741165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3950.565124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3950.570692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3950.578303] LustreError: Skipped 1 previous similar message [ 3950.579282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3950.587061] Lustre: Skipped 1 previous similar message [ 3950.612703] Lustre: Skipped 1 previous similar message [ 3954.881530] Lustre: server umount lustre-OST0000 complete [ 3959.779581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.784155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.793682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3960.803460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3960.809977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.883890] Lustre: server umount lustre-MDT0000 complete [ 3968.680729] LustreError: 136248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448070 with bad export cookie 5388768583877395546 [ 3968.686951] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.967955] Lustre: server umount lustre-MDT0001 complete [ 3977.570357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.170040] LustreError: 138467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3978.187524] LustreError: 138467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3978.261801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.119677] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 3983.340736] LustreError: 138468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.452437] LustreError: 138467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.882439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3992.245836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3996.429559] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4002.981124] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4007.898193] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4013.715723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4014.002512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4019.678769] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4025.893448] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4029.462776] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4029.646190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4033.654191] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4033.854958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4041.954792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4042.134208] Lustre: lustre-OST0001: new disk, initializing [ 4042.142933] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4042.153548] Lustre: 141837:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4042.265718] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4048.164831] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4049.017187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4049.031527] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4049.199979] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4055.459184] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4055.735856] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4060.712758] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4061.016126] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4067.444794] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92116017e800.ost_server_uuid 50 [ 4069.137732] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92116017e800.ost_server_uuid in FULL state after 0 sec [ 4079.854176] Lustre: server umount lustre-OST0001 complete [ 4080.097331] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4080.107190] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.125650] LustreError: 140290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4080.128773] Lustre: Skipped 1 previous similar message [ 4080.149987] LustreError: 140290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4083.683487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4083.691309] LustreError: Skipped 1 previous similar message [ 4083.697633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.705395] Lustre: Skipped 1 previous similar message [ 4084.706803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.716659] Lustre: Skipped 1 previous similar message [ 4085.222762] LustreError: 140290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4085.239066] LustreError: 140290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4089.883416] Lustre: server umount lustre-OST0000 complete [ 4094.947284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4094.958177] LustreError: Skipped 1 previous similar message [ 4094.964301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4094.985847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4094.999661] Lustre: Skipped 3 previous similar messages [ 4099.335294] Lustre: server umount lustre-MDT0000 complete [ 4100.577747] LustreError: 138467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4100.593347] LustreError: 138467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4102.699968] LustreError: 138492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448204 with bad export cookie 5388768583877396428 [ 4102.705166] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4102.718698] LustreError: 138492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4103.332408] Lustre: server umount lustre-MDT0001 complete [ 4120.196436] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 4122.578919] Key type lgssc unregistered [ 4122.806753] LNet: 144841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4122.812953] LNetError: 144841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4123.880268] LNet: Removed LNI 192.168.201.115@tcp [ 4124.589409] Key type .llcrypt unregistered [ 4124.591261] Key type ._llcrypt unregistered [ 4161.771944] Key type ._llcrypt registered [ 4161.777159] Key type .llcrypt registered [ 4161.928894] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 4176.075655] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 4177.483433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4177.536181] alg: No test for adler32 (adler32-zlib) [ 4178.637195] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4178.824046] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 4180.481089] Key type lgssc registered [ 4181.735513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4214.302653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4214.344580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4215.579862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4215.606633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4215.664257] Lustre: lustre-MDT0000: new disk, initializing [ 4215.721106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.737608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4221.334555] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4230.957630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4231.094979] Lustre: 148997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4231.190675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4231.203406] Lustre: Skipped 1 previous similar message [ 4231.300868] Lustre: lustre-MDT0001: new disk, initializing [ 4231.371114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4231.409098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4231.429782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4235.548713] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4244.061221] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4249.484689] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4256.752502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4257.099448] Lustre: lustre-OST0000: new disk, initializing [ 4257.103569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4257.109360] Lustre: 150964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4257.195190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.067148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4264.082285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4264.249534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4264.649905] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4277.377600] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4282.115360] Lustre: Failing over lustre-OST0000 [ 4282.611716] Lustre: server umount lustre-OST0000 complete [ 4300.192828] Lustre: 147223:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448385/real 1788448385] req@000000001aba8e0a x1875323943684096/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788448401 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4300.217399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.288159] Lustre: 147223:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448386/real 1788448386] req@0000000067ce0919 x1875323943684224/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788448402 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4301.289804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.315212] Lustre: 147223:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4302.493258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4303.107258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.163975] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4306.155525] Lustre: 147222:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448391/real 1788448391] req@0000000020dd1747 x1875323943686144/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788448407 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4306.217713] Lustre: 147222:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4310.251980] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4310.496287] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4311.371710] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4311.377592] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4312.416122] Lustre: 147222:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448396/real 1788448396] req@0000000021399010 x1875323943687296/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788448412 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4315.775050] Lustre: Failing over lustre-MDT0000 [ 4316.577909] Lustre: 147222:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448401/real 1788448401] req@0000000063fe6dae x1875323943688320/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788448417 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4316.605622] Lustre: 147222:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4318.221602] Lustre: server umount lustre-MDT0000 complete [ 4318.693057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4318.703600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4320.747148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4320.764305] Lustre: Skipped 1 previous similar message [ 4325.845826] LustreError: 149004:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.868649] LustreError: 149004:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 4330.963756] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4330.989601] LustreError: 149003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4336.086724] LustreError: 152883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4336.112800] LustreError: 152883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4336.821455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.011148] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4337.244724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.291945] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4341.226558] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4342.239806] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4342.271676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4342.283774] Lustre: Skipped 1 previous similar message [ 4342.309273] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4342.365528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4352.481897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4352.489032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4352.491562] Lustre: Skipped 1 previous similar message [ 4352.505172] Lustre: Skipped 1 previous similar message [ 4357.604483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4357.618125] Lustre: Skipped 1 previous similar message [ 4362.738837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4362.751840] Lustre: Skipped 1 previous similar message [ 4363.232401] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4363.557540] Lustre: server umount lustre-OST0000 complete [ 4367.842054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.851272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4367.879203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4367.897421] Lustre: Skipped 1 previous similar message [ 4372.962336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4372.974323] Lustre: Skipped 1 previous similar message [ 4373.471214] Lustre: server umount lustre-MDT0000 complete [ 4377.127370] LustreError: 150974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448478 with bad export cookie 15229696108928065993 [ 4377.143094] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4377.146680] LustreError: 150974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4377.554702] Lustre: server umount lustre-MDT0001 complete [ 4387.234136] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 4390.070218] Key type lgssc unregistered [ 4390.382659] LNet: 154638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4390.391132] LNetError: 154638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4390.410039] LNet: Removed LNI 192.168.201.115@tcp [ 4391.258102] Key type .llcrypt unregistered [ 4391.261952] Key type ._llcrypt unregistered [ 4426.415934] Key type ._llcrypt registered [ 4426.419295] Key type .llcrypt registered [ 4426.547439] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 4427.736983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4427.865410] alg: No test for adler32 (adler32-zlib) [ 4428.872245] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4429.083232] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 4430.728172] Key type lgssc registered [ 4431.750646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4438.801142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4438.817501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.142854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4444.019495] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4450.277261] LustreError: 156156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.304606] LustreError: 156156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4451.938129] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4455.401782] LustreError: 156157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4455.608303] LustreError: 156185:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4455.617356] LustreError: 156185:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4455.626417] LustreError: 156185:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4455.837866] Lustre: server umount lustre-MDT0000 complete [ 4469.188689] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 4478.495165] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 4509.243252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.468913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4509.507696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4509.579601] Lustre: lustre-MDT0000: new disk, initializing [ 4509.652910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.676544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4513.664788] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4523.130921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.272661] Lustre: 159487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4523.345588] LustreError: 159497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.362992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4523.367854] Lustre: Skipped 1 previous similar message [ 4523.413815] Lustre: lustre-MDT0001: new disk, initializing [ 4523.496964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4523.526660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4523.537917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4528.749813] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4537.381750] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4543.927605] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4551.073630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4551.271180] Lustre: lustre-OST0000: new disk, initializing [ 4551.274959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4551.281246] Lustre: 161455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4551.393433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4556.700113] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4558.918798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4558.928151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4559.011283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4565.312119] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4568.775895] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4570.476561] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:17:50 (1788448670) [ 4769.249647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4769.250414] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4769.265961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4769.275250] Lustre: Skipped 2 previous similar messages [ 4769.290918] Lustre: Skipped 2 previous similar messages [ 4772.512625] Lustre: server umount lustre-MDT0000 complete [ 4774.370410] LustreError: 162727:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.380819] LustreError: 162727:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4776.282443] LustreError: 159478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448877 with bad export cookie 2038180121371221464 [ 4776.294550] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4776.295669] LustreError: 159478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4779.490944] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4779.496698] LustreError: 161087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4779.519389] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4781.543689] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4782.700657] Lustre: server umount lustre-MDT0001 complete [ 4793.466996] Lustre: server umount lustre-OST0000 complete [ 4805.188706] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 4814.575407] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 4849.064765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4849.271342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4849.296355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4849.358266] Lustre: lustre-MDT0000: new disk, initializing [ 4849.424923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4849.436580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4853.432376] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4862.454717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.590717] Lustre: 166114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4862.645744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4862.649789] Lustre: Skipped 1 previous similar message [ 4862.727771] Lustre: lustre-MDT0001: new disk, initializing [ 4862.841730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4862.903068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4862.917419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4866.761666] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4875.888308] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4881.358338] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4888.870120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4889.106926] Lustre: lustre-OST0000: new disk, initializing [ 4889.116626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4889.127222] Lustre: 168080:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4889.251886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4895.766155] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 4897.395783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4897.413773] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4897.520793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4904.492653] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4909.804554] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4910.059293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4915.377107] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4915.646645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4918.243054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4918.257434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.269153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4918.284359] Lustre: Skipped 1 previous similar message [ 4924.115739] Lustre: server umount lustre-OST0000 complete [ 4929.507340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4929.513617] LustreError: Skipped 1 previous similar message [ 4929.521295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4929.533035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4929.536221] Lustre: Skipped 3 previous similar messages [ 4934.831710] Lustre: server umount lustre-MDT0000 complete [ 4936.682734] LustreError: 166119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.725703] LustreError: 166119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4938.564401] LustreError: 166105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449039 with bad export cookie 2038180121371380287 [ 4938.566786] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4938.572607] LustreError: 166105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4938.954734] Lustre: server umount lustre-MDT0001 complete [ 4948.454150] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:24:08 (1788449048) [ 4963.299705] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 4971.129258] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5005.758838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5006.076847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5006.108380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5006.182400] Lustre: lustre-MDT0000: new disk, initializing [ 5006.307563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5006.327472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5011.533876] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5021.585960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5021.690197] Lustre: 172834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5021.903248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5025.615984] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5034.856374] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5040.353695] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5047.032652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5047.362314] Lustre: lustre-OST0000: new disk, initializing [ 5047.366458] Lustre: Skipped 1 previous similar message [ 5047.369334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5047.372841] Lustre: Skipped 2 previous similar messages [ 5047.376957] Lustre: 174808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5053.551133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5053.565177] Lustre: Skipped 1 previous similar message [ 5053.574440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5053.764744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5054.364314] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5064.641895] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5072.659632] Lustre: server umount lustre-OST0000 complete [ 5078.497571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5078.514543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5078.531115] Lustre: Skipped 1 previous similar message [ 5078.540407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5078.545542] Lustre: Skipped 2 previous similar messages [ 5086.844260] Lustre: server umount lustre-MDT0000 complete [ 5089.760149] Lustre: 155467:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449175/real 1788449175] req@0000000051d2d595 x1875324206545920/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788449191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5089.774870] LustreError: 175793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5089.778410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.778420] Lustre: Skipped 1 previous similar message [ 5089.811237] Lustre: 155467:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5089.831368] LustreError: 175793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5090.214683] LustreError: 172826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449191 with bad export cookie 2038180121371381176 [ 5090.215497] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5090.220655] LustreError: 172826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5090.500435] Lustre: server umount lustre-MDT0001 complete [ 5097.330350] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:26:37 (1788449197) [ 5112.201438] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 5121.724988] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5165.009055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5165.261923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5165.303434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5165.457143] Lustre: lustre-MDT0000: new disk, initializing [ 5165.531406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5165.538247] Lustre: Skipped 2 previous similar messages [ 5165.548989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5171.027525] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5181.880620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.965971] Lustre: 179131:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5182.240752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5186.498238] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5194.891561] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5200.814701] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5207.227145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5207.514501] Lustre: 181111:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5212.733884] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5214.282903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5214.398687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5220.572732] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5230.049670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5230.049684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5230.059536] LustreError: Skipped 1 previous similar message [ 5230.061354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5230.061360] Lustre: Skipped 4 previous similar messages [ 5230.104448] Lustre: Skipped 1 previous similar message [ 5232.613366] Lustre: server umount lustre-OST0000 complete [ 5238.754184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5242.807702] Lustre: server umount lustre-MDT0000 complete [ 5245.419833] LustreError: 180447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.445535] LustreError: 180447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5247.664497] LustreError: 179124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449348 with bad export cookie 2038180121371382261 [ 5247.671971] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5247.673214] LustreError: 179124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5247.914126] Lustre: server umount lustre-MDT0001 complete [ 5260.011957] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 5262.862165] Key type lgssc unregistered [ 5263.164760] LNet: 182969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5263.175478] LNetError: 182969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5263.215314] LNet: Removed LNI 192.168.201.115@tcp [ 5264.345523] Key type .llcrypt unregistered [ 5264.350408] Key type ._llcrypt unregistered [ 5287.133885] Key type ._llcrypt registered [ 5287.136425] Key type .llcrypt registered [ 5287.237948] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5288.317475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5288.416663] alg: No test for adler32 (adler32-zlib) [ 5289.649135] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5289.901182] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 5291.588169] Key type lgssc registered [ 5292.903730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5300.803376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5300.836704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5302.500901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5307.093478] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5311.978303] LustreError: 184385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5312.011187] LustreError: 184385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5316.114600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.608137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5321.327342] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5328.543685] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5334.878477] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5341.633700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5342.053759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.185068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5347.928386] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5354.930823] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5376.999122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5376.999375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5377.001723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5377.029703] Lustre: Skipped 1 previous similar message [ 5381.644935] Lustre: server umount lustre-OST0000 complete [ 5387.237076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.248573] Lustre: Skipped 1 previous similar message [ 5387.257647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5387.268200] Lustre: Skipped 1 previous similar message [ 5391.596850] Lustre: server umount lustre-MDT0000 complete [ 5392.363285] LustreError: 184385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5395.319379] LustreError: 184367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449496 with bad export cookie 17345750937474529610 [ 5395.352741] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5395.625573] Lustre: server umount lustre-MDT0001 complete [ 5406.088300] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 5408.636039] Key type lgssc unregistered [ 5408.967878] LNet: 187943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5408.980699] LNetError: 187943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5409.008609] LNet: Removed LNI 192.168.201.115@tcp [ 5409.789516] Key type .llcrypt unregistered [ 5409.791326] Key type ._llcrypt unregistered [ 5436.017738] Key type ._llcrypt registered [ 5436.023362] Key type .llcrypt registered [ 5436.231497] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5438.596317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5438.647172] alg: No test for adler32 (adler32-zlib) [ 5439.641658] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5439.861060] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 5441.536409] Key type lgssc registered [ 5442.768996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5451.582639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5451.609128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5453.159275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.024417] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5463.530322] LustreError: 189364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.547748] LustreError: 189364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5467.333988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.794225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5472.470233] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5479.926506] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5487.055824] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5496.175509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5496.734208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5497.806290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5503.244799] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5509.966379] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5522.945662] Lustre: lustre-OST0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 5523.427290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5523.427534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.437846] LustreError: Skipped 1 previous similar message [ 5523.447672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5523.453695] Lustre: Skipped 1 previous similar message [ 5525.761227] Lustre: server umount lustre-OST0000 complete [ 5531.586427] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff921146f62000.ost_server_uuid 50 [ 5533.077269] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5548.005143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5548.008473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5548.013295] Lustre: Skipped 2 previous similar messages [ 5548.025153] Lustre: Skipped 1 previous similar message [ 5553.125661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5553.129904] Lustre: Skipped 1 previous similar message [ 5553.715335] Lustre: server umount lustre-MDT0000 complete [ 5557.050571] LustreError: 189346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449658 with bad export cookie 3233463406003882193 [ 5557.059831] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5557.061495] LustreError: 189346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5557.584666] Lustre: server umount lustre-MDT0001 complete [ 5567.199741] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:34:27 (1788449667) [ 5572.819329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5573.262732] LustreError: 193287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.372278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5578.036713] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5578.726636] LustreError: 193288:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5582.852453] LustreError: 193287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.571694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.142127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5591.958770] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5600.589839] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5606.975937] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5618.571739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5619.177000] LustreError: 195106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5619.456737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5624.865044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5628.852385] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5637.057677] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5645.360807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5645.763561] Lustre: lustre-OST0001: new disk, initializing [ 5645.770416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5645.788818] Lustre: 196130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5645.959733] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5651.654472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5651.686781] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5651.911569] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5653.289895] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5664.588219] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5667.810540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5667.824379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.850222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5670.884294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5670.901710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.775662] Lustre: server umount lustre-OST0000 complete [ 5675.486647] LustreError: 195767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5680.665737] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5680.869597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5685.781771] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5686.016444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5692.907292] LustreError: 195109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5692.921578] LustreError: 195109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5703.142730] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.171730] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5703.177379] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5703.189545] LustreError: Skipped 1 previous similar message [ 5703.217047] Lustre: Skipped 5 previous similar messages [ 5707.277933] Lustre: server umount lustre-OST0001 complete [ 5713.380086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5713.403342] Lustre: Skipped 2 previous similar messages [ 5718.398672] Lustre: server umount lustre-MDT0000 complete [ 5722.707139] LustreError: 193268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449824 with bad export cookie 3233463406003883264 [ 5722.714605] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5722.721583] LustreError: 193268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5723.457466] Lustre: server umount lustre-MDT0001 complete [ 5750.900171] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:37:30 (1788449850) [ 5756.740584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5757.099428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5757.194640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5757.442954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5761.485237] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5768.594671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.663030] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5768.694913] Lustre: 199564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5772.530243] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5779.879856] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5785.673935] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5793.559443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5793.874394] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5801.633849] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5803.731108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5809.441736] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5815.629350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5815.678588] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5822.115382] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5829.145992] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5834.222762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5834.224514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5834.228799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5834.233077] Lustre: Skipped 1 previous similar message [ 5834.248167] Lustre: Skipped 3 previous similar messages [ 5837.488795] Lustre: server umount lustre-OST0000 complete [ 5839.341133] LustreError: 201416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5839.359860] LustreError: 201416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5854.188122] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5859.800263] Lustre: server umount lustre-OST0001 complete [ 5870.053112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5870.059184] Lustre: Skipped 6 previous similar messages [ 5870.405267] Lustre: server umount lustre-MDT0000 complete [ 5873.687915] LustreError: 199557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449975 with bad export cookie 3233463406003884664 [ 5873.693837] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5873.699433] LustreError: 199557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5874.226751] Lustre: server umount lustre-MDT0001 complete [ 5897.208853] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:39:57 (1788449997) [ 5914.063317] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 5925.286320] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 5965.595873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.937411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5965.968756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5966.044610] Lustre: lustre-MDT0000: new disk, initializing [ 5966.123472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5966.129820] Lustre: Skipped 3 previous similar messages [ 5966.143879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5971.202428] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5981.816084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5981.976914] Lustre: 207661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5982.034414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5982.039194] Lustre: Skipped 1 previous similar message [ 5982.115409] Lustre: lustre-MDT0001: new disk, initializing [ 5982.232221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5982.256545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5986.824704] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 5996.752506] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6003.042044] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6008.992138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6009.265439] Lustre: lustre-OST0000: new disk, initializing [ 6009.268356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6009.273203] Lustre: 209629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6015.037665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6015.053256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6015.167760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6015.282648] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6025.568698] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6031.488594] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6031.813522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6038.315525] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6038.589737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6041.063770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.076595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6041.084444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6041.084454] LustreError: Skipped 1 previous similar message [ 6041.088770] Lustre: Skipped 5 previous similar messages [ 6041.132656] Lustre: Skipped 1 previous similar message [ 6046.604980] Lustre: server umount lustre-OST0000 complete [ 6065.120572] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6065.317782] Lustre: server umount lustre-MDT0000 complete [ 6066.657647] LustreError: 210176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6066.690931] LustreError: 210176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6068.710672] LustreError: 209628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450170 with bad export cookie 3233463406003886232 [ 6068.713951] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6068.728642] LustreError: 209628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6069.107450] Lustre: server umount lustre-MDT0001 complete [ 6076.248721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6080.626340] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6090.065112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.388528] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6100.578986] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6106.543385] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6112.090988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6112.332492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6112.343704] Lustre: Skipped 4 previous similar messages [ 6119.029386] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6127.876050] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6134.167665] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6134.448880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6139.262324] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6139.460145] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6141.411207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6141.415940] LustreError: Skipped 1 previous similar message [ 6141.418923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6141.429123] Lustre: Skipped 2 previous similar messages [ 6147.537500] Lustre: server umount lustre-OST0000 complete [ 6153.936454] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6159.225978] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6187.995838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6192.369480] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6192.399878] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6195.753836] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6203.526231] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6206.763845] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6206.780732] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6207.174134] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6212.069303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6212.078795] Lustre: Skipped 12 previous similar messages [ 6213.047492] Lustre: server umount lustre-OST0000 complete [ 6223.187231] Lustre: server umount lustre-MDT0000 complete [ 6227.334052] LustreError: 211831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450328 with bad export cookie 3233463406003887114 [ 6227.341463] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6227.353567] LustreError: 211831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6227.817381] Lustre: server umount lustre-MDT0001 complete [ 6235.829189] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:45:35 (1788450335) [ 6244.584640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.005505] LustreError: 217817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6245.032935] LustreError: 217817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6250.471668] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6258.744902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.580350] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6270.380690] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6275.936140] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6281.966621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6288.244717] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6296.128257] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6300.575705] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6300.778706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6305.266635] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6305.475265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6312.101241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6312.285540] Lustre: lustre-OST0001: new disk, initializing [ 6312.289211] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6312.296610] Lustre: 221188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6318.943824] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6319.312694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6319.324565] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6319.446714] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6328.195392] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6332.737877] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6332.930208] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6336.791069] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6336.953386] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6340.068828] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.074876] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6340.075095] Lustre: Skipped 5 previous similar messages [ 6340.097472] LustreError: Skipped 1 previous similar message [ 6344.890935] Lustre: server umount lustre-OST0001 complete [ 6350.299944] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6350.482861] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6354.461312] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6354.627430] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6383.899678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6384.166329] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6384.174460] Lustre: Skipped 5 previous similar messages [ 6389.237907] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6389.263249] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6389.264111] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6389.684657] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6395.993393] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6397.472758] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6405.366261] Lustre: server umount lustre-OST0001 complete [ 6435.297139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6435.696970] Lustre: server umount lustre-OST0000 complete [ 6446.922346] Lustre: server umount lustre-MDT0000 complete [ 6450.730144] LustreError: 217799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450552 with bad export cookie 3233463406003888080 [ 6450.737370] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6450.743470] LustreError: 217799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6457.240708] Lustre: server umount lustre-MDT0001 complete [ 6478.454341] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:49:38 (1788450578) [ 6484.696599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6484.941811] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6484.972864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6489.235331] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6498.019437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6498.099680] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6498.128983] Lustre: 226431:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6503.658235] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6510.689928] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6515.265919] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6521.311453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6521.440900] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6526.874463] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6534.140207] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6543.159750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6543.212096] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6543.327813] LustreError: 228278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6543.360031] LustreError: 228278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 40 previous similar messages [ 6550.741868] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6558.002802] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6559.445068] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6563.323348] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6563.616593] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6567.860360] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6568.126417] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6574.254245] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9211435dc000.ost_server_uuid 50 [ 6575.782966] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9211435dc000.ost_server_uuid in IDLE state after 0 sec [ 6576.708251] Lustre: Permanently deactivating lustre-OST0001 [ 6576.727448] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6580.259535] Lustre: Permanently reactivating lustre-OST0001 [ 6587.875178] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6587.881804] LustreError: Skipped 4 previous similar messages [ 6587.887808] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6587.893622] Lustre: Skipped 23 previous similar messages [ 6590.601869] Lustre: server umount lustre-OST0001 complete [ 6600.763555] Lustre: server umount lustre-OST0000 complete [ 6605.798310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6605.812320] Lustre: Skipped 11 previous similar messages [ 6611.991964] Lustre: server umount lustre-MDT0000 complete [ 6615.603274] LustreError: 226423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450716 with bad export cookie 3233463406003889368 [ 6615.604735] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6615.616667] LustreError: 226423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6616.005615] Lustre: server umount lustre-MDT0001 complete [ 6638.604565] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:52:18 (1788450738) [ 6645.774636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6646.202105] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6646.270760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6646.293144] Lustre: Skipped 5 previous similar messages [ 6651.116944] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6659.168659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.268869] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6659.296923] Lustre: 233122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6662.972970] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6669.086678] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6674.064907] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6679.720323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6680.016437] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6680.067716] Lustre: Permanently deactivating lustre-OST0000 [ 6680.090443] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6685.254288] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6687.265554] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6691.899901] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6698.889708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6699.041126] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6707.574606] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6708.102060] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6715.592572] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6717.877591] Lustre: Permanently reactivating lustre-OST0000 [ 6717.891433] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6717.897752] Lustre: Skipped 2 previous similar messages [ 6727.739825] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6727.746495] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 000000002cc58fb9, cur 1788450829 deadline 1788450818 last 1788450788 [ 6727.803941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6733.093470] Lustre: server umount lustre-OST0001 complete [ 6742.741620] Lustre: server umount lustre-OST0000 complete [ 6752.328426] Lustre: server umount lustre-MDT0000 complete [ 6755.337956] LustreError: 233113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450856 with bad export cookie 3233463406003891013 [ 6755.341739] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6755.347708] LustreError: 233113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6755.640165] Lustre: server umount lustre-MDT0001 complete [ 6766.181364] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 6769.074965] Key type lgssc unregistered [ 6769.384658] LNet: 238001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6769.399714] LNetError: 238001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6770.472902] LNet: Removed LNI 192.168.201.115@tcp [ 6771.424055] Key type .llcrypt unregistered [ 6771.425689] Key type ._llcrypt unregistered [ 6795.762226] Key type ._llcrypt registered [ 6795.766563] Key type .llcrypt registered [ 6795.870853] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 6797.261286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6797.345818] alg: No test for adler32 (adler32-zlib) [ 6798.387396] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6798.684945] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 6800.320233] Key type lgssc registered [ 6801.550779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6810.286831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6810.305529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6811.683642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6816.509035] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6820.860838] LustreError: 239469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6820.890410] LustreError: 239469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6823.878708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6823.984436] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6824.020606] Lustre: 239458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6824.030446] Lustre: Permanently deactivating lustre-MDT0001 [ 6824.042385] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6824.330889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6826.966957] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6826.970539] LustreError: 239500:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6826.976510] LustreError: 239500:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6826.994684] LustreError: 239500:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6828.241703] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6836.008882] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6841.303777] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6847.841255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6848.201957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6849.251651] LustreError: 241311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6854.171975] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6854.627579] LustreError: 241311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.676574] LustreError: 241398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.687069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6856.699885] LustreError: 241398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6861.221597] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6861.812080] LustreError: 241398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6861.826798] LustreError: 241398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6868.063537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6868.351681] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6873.593350] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6873.596432] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6874.272210] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 6881.546138] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6885.135616] Lustre: Permanently reactivating lustre-MDT0001 [ 6885.150345] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6885.156424] Lustre: Skipped 2 previous similar messages [ 6888.612351] Lustre: lustre-MDT0001: Client 28c54e54-00f4-4cd1-978c-4d1bb2f67dc2 (at 192.168.201.15@tcp) reconnecting [ 6894.733414] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6896.221262] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6900.907320] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6901.114339] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6904.880196] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6905.052441] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6906.918911] Lustre: Permanently deactivating lustre-MDT0001 [ 6906.944294] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6906.958092] Lustre: Skipped 2 previous similar messages [ 6914.159136] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6921.700140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6921.711905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.736030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6924.772897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6924.776952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6924.782508] Lustre: Skipped 2 previous similar messages [ 6924.795273] Lustre: Skipped 3 previous similar messages [ 6926.109427] Lustre: server umount lustre-MDT0000 complete [ 6929.614200] LustreError: 239449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451030 with bad export cookie 7886588193342603095 [ 6929.625249] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6929.631648] LustreError: 239449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6929.894114] LustreError: 244090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.923549] LustreError: 244090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6930.021399] Lustre: server umount lustre-MDT0001 complete [ 6935.905750] Lustre: server umount lustre-OST0000 complete [ 6946.109299] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:57:26 (1788451046) [ 6955.618461] Lustre: server umount lustre-OST0001 complete [ 6962.338359] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 6971.852572] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7011.030204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7011.248676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7011.255852] Lustre: Skipped 2 previous similar messages [ 7011.319483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7011.388477] Lustre: lustre-MDT0000: new disk, initializing [ 7011.520700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7011.543959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7015.308106] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7025.511670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.631112] Lustre: 247727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7025.698969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7025.707152] Lustre: Skipped 1 previous similar message [ 7025.789526] Lustre: lustre-MDT0001: new disk, initializing [ 7025.925401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7025.962936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7025.979457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7030.607190] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7039.097670] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7044.554134] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7051.278490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7051.562117] Lustre: lustre-OST0000: new disk, initializing [ 7051.567542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7051.578462] Lustre: 249693:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7051.660835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7058.402850] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7061.624478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7061.659390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7061.844731] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7069.531619] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7072.330656] LustreError: 247733:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7082.031560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7082.218946] Lustre: lustre-OST0001: new disk, initializing [ 7082.225372] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7082.233308] Lustre: 250808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7082.424206] LustreError: 247733:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7082.459346] LustreError: 249528:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7088.214739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7088.232183] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7088.317143] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7088.747539] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7092.528615] LustreError: 249528:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7097.040360] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.382552] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7101.914703] Lustre: lustre-OST0001: Not available for connect from 192.168.201.15@tcp (stopping) [ 7103.970130] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7103.970755] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.978308] LustreError: Skipped 1 previous similar message [ 7103.984237] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7104.016891] Lustre: Skipped 1 previous similar message [ 7106.332078] Lustre: server umount lustre-OST0001 complete [ 7107.028512] LustreError: 249679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7107.048959] LustreError: 249679:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7123.430068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7123.434933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7123.441585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7123.449076] Lustre: Skipped 1 previous similar message [ 7123.456662] Lustre: Skipped 2 previous similar messages [ 7128.676504] Lustre: server umount lustre-OST0000 complete [ 7132.647409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7132.666202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7132.671921] Lustre: Skipped 1 previous similar message [ 7132.696955] Lustre: Skipped 3 previous similar messages [ 7138.376371] Lustre: server umount lustre-MDT0000 complete [ 7141.636870] LustreError: 247718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451242 with bad export cookie 7886588193342605692 [ 7141.637921] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7141.649920] LustreError: 247718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7148.096649] Lustre: server umount lustre-MDT0001 complete [ 7158.987561] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7162.386922] Key type lgssc unregistered [ 7162.791219] LNet: 252859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7162.801983] LNetError: 252859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7162.823191] LNet: Removed LNI 192.168.201.115@tcp [ 7163.790138] Key type .llcrypt unregistered [ 7163.793553] Key type ._llcrypt unregistered [ 7205.084593] Key type ._llcrypt registered [ 7205.086323] Key type .llcrypt registered [ 7205.165824] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7206.167070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7206.370951] alg: No test for adler32 (adler32-zlib) [ 7207.428258] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7207.636514] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7209.352189] Key type lgssc registered [ 7210.651721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7218.076837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7218.094166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7219.341812] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7219.358268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7219.477879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7224.293676] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7234.195372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7234.246990] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7234.270593] Lustre: 255069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7234.715715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7238.972379] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7246.626386] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7253.082389] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7259.086786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7259.278796] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7259.569148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7265.788982] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7268.486272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7273.054747] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7324.641762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7324.652811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.674052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.166609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.175535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7325.182136] Lustre: Skipped 1 previous similar message [ 7325.184448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7327.506554] Lustre: server umount lustre-OST0000 complete [ 7330.987158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7341.450580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7341.928996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7345.012391] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7345.036057] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7347.456470] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7350.213623] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7350.235040] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7353.943506] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7396.321338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7396.324530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7396.332986] LustreError: Skipped 1 previous similar message [ 7396.336733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7396.353233] Lustre: Skipped 1 previous similar message [ 7396.378466] Lustre: Skipped 1 previous similar message [ 7398.067716] Lustre: server umount lustre-OST0000 complete [ 7403.489525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7403.493266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7403.509912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7411.682023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7411.690660] Lustre: Skipped 3 previous similar messages [ 7416.288198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7416.484980] Lustre: server umount lustre-MDT0000 complete [ 7419.934208] LustreError: 255060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451521 with bad export cookie 12982419325208698968 [ 7419.942991] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7419.946278] LustreError: 255060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7420.270732] Lustre: server umount lustre-MDT0001 complete [ 7431.150959] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7433.953527] Key type lgssc unregistered [ 7434.268323] LNet: 260494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7434.276946] LNetError: 260494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7435.311341] LNet: Removed LNI 192.168.201.115@tcp [ 7436.206450] Key type .llcrypt unregistered [ 7436.208351] Key type ._llcrypt unregistered [ 7460.500374] Key type ._llcrypt registered [ 7460.502297] Key type .llcrypt registered [ 7460.594532] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7461.966815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7462.012426] alg: No test for adler32 (adler32-zlib) [ 7463.076890] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7463.304983] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7464.905603] Key type lgssc registered [ 7466.170960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7474.122870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7474.137432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.719992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.727659] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7484.900718] LustreError: 261910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7484.929102] LustreError: 261910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7489.275908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7489.511898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7493.241560] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7500.212554] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7505.126525] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7511.687323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.017065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.971770] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7520.187924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7525.543607] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7549.922884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7549.935254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7549.961150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7550.957549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7550.970614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.981871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7551.003189] Lustre: Skipped 1 previous similar message [ 7553.548598] Lustre: server umount lustre-OST0000 complete [ 7561.187900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.195059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7561.195242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7561.199666] Lustre: Skipped 1 previous similar message [ 7563.934356] Lustre: server umount lustre-MDT0000 complete [ 7566.318481] LustreError: 261910:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.335970] LustreError: 261910:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7567.765606] LustreError: 261892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451669 with bad export cookie 5128714938356797630 [ 7567.778575] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7567.784211] LustreError: 261892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7568.196250] Lustre: server umount lustre-MDT0001 complete [ 7579.224145] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7581.815864] Key type lgssc unregistered [ 7582.083666] LNet: 266217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7582.096859] LNetError: 266217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7583.144819] LNet: Removed LNI 192.168.201.115@tcp [ 7584.319265] Key type .llcrypt unregistered [ 7584.322596] Key type ._llcrypt unregistered [ 7601.699555] Key type ._llcrypt registered [ 7601.704433] Key type .llcrypt registered [ 7601.806149] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7602.751448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7602.870180] alg: No test for adler32 (adler32-zlib) [ 7604.056416] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7604.438343] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7606.168208] Key type lgssc registered [ 7607.097388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7615.047723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7615.068745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7616.437318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7621.236330] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7625.733814] LustreError: 267441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7625.750997] LustreError: 267441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7629.580581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.026937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7634.697151] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7642.002240] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7647.408417] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7653.460739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7653.891468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7660.490458] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7661.125430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7667.608513] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7680.481445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7680.484581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7680.525125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7681.506728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7681.510937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7681.523762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.415421] Lustre: server umount lustre-OST0000 complete [ 7691.748346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7691.761445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7691.766059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7691.768868] Lustre: Skipped 1 previous similar message [ 7691.775764] Lustre: Skipped 2 previous similar messages [ 7693.273231] Lustre: server umount lustre-MDT0000 complete [ 7696.867611] LustreError: 267438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.893566] LustreError: 267438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7697.104425] LustreError: 267423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451798 with bad export cookie 2974080743840640145 [ 7697.109398] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7697.113355] LustreError: 267423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7697.484509] Lustre: server umount lustre-MDT0001 complete [ 7708.582233] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7711.512922] Key type lgssc unregistered [ 7711.817599] LNet: 271121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7711.821733] LNetError: 271121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7711.850179] LNet: Removed LNI 192.168.201.115@tcp [ 7712.606078] Key type .llcrypt unregistered [ 7712.608577] Key type ._llcrypt unregistered [ 7737.169996] Key type ._llcrypt registered [ 7737.173220] Key type .llcrypt registered [ 7737.276565] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7738.547422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7738.630161] alg: No test for adler32 (adler32-zlib) [ 7739.916373] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7740.214260] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7741.952171] Key type lgssc registered [ 7743.052082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7751.056205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7751.087939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7752.605951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7756.848897] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7761.900441] LustreError: 272538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.914959] LustreError: 272538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7765.546599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.938179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7770.333797] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7778.826508] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7787.002598] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7794.591685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7794.940597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.983295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7802.137736] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7809.996924] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7837.158088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7837.163627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.165429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7837.191470] Lustre: Skipped 1 previous similar message [ 7839.579814] Lustre: server umount lustre-OST0000 complete [ 7847.397508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7847.413982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7847.420018] Lustre: Skipped 2 previous similar messages [ 7852.513806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7852.524102] Lustre: Skipped 1 previous similar message [ 7857.634278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7857.643531] Lustre: Skipped 1 previous similar message [ 7858.146153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7858.394532] Lustre: server umount lustre-MDT0000 complete [ 7862.098041] LustreError: 272519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451963 with bad export cookie 10074470773537854352 [ 7862.110354] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7862.126486] LustreError: 272519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7862.734853] Lustre: server umount lustre-MDT0001 complete [ 7872.412839] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 7875.283983] Key type lgssc unregistered [ 7875.553883] LNet: 276849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7875.570913] LNetError: 276849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7875.602025] LNet: Removed LNI 192.168.201.115@tcp [ 7876.666877] Key type .llcrypt unregistered [ 7876.671751] Key type ._llcrypt unregistered [ 7894.848957] Key type ._llcrypt registered [ 7894.853827] Key type .llcrypt registered [ 7894.999917] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 7896.204386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7896.275132] alg: No test for adler32 (adler32-zlib) [ 7897.302781] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7897.555811] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 7899.216240] Key type lgssc registered [ 7900.419551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7907.557341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7907.586925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.346996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7914.608353] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7919.590177] LustreError: 278085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7919.599365] LustreError: 278085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7923.770080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7924.124161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7929.082364] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7936.461246] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7942.329946] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7949.513676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7949.899354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.056340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7956.671663] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 7962.833045] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7980.006447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7980.008710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7980.015571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7980.038618] Lustre: Skipped 1 previous similar message [ 7983.832497] Lustre: server umount lustre-OST0000 complete [ 7990.244177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.261129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.267673] Lustre: Skipped 1 previous similar message [ 7990.289752] Lustre: Skipped 2 previous similar messages [ 7994.624279] Lustre: server umount lustre-MDT0000 complete [ 7995.364855] LustreError: 278077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7995.376990] LustreError: 278077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7998.228920] LustreError: 278050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452099 with bad export cookie 7500463749431453502 [ 7998.237079] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7998.251619] LustreError: 278050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7998.766672] Lustre: server umount lustre-MDT0001 complete [ 8011.664391] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 8014.330130] Key type lgssc unregistered [ 8014.588760] LNet: 281792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8014.591701] LNetError: 281792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8014.624762] LNet: Removed LNI 192.168.201.115@tcp [ 8015.392690] Key type .llcrypt unregistered [ 8015.394549] Key type ._llcrypt unregistered [ 8027.555069] Key type ._llcrypt registered [ 8027.557316] Key type .llcrypt registered [ 8027.653936] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8044.863316] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 8059.967304] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8060.928740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8061.063945] alg: No test for adler32 (adler32-zlib) [ 8062.075585] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8062.305420] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 8064.024438] Key type lgssc registered [ 8065.206632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8100.577589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8100.602297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8101.863321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8101.939037] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8102.030636] Lustre: lustre-MDT0000: new disk, initializing [ 8102.102222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8102.124135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8107.659656] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8118.712472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8118.880159] Lustre: 285343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8118.931339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8118.934944] Lustre: Skipped 1 previous similar message [ 8119.035083] Lustre: lustre-MDT0001: new disk, initializing [ 8119.162497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8119.195639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8119.206747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8123.961704] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8134.154762] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8140.536467] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8147.632654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8147.935601] Lustre: lustre-OST0000: new disk, initializing [ 8147.941327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8147.947611] Lustre: 287313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8148.061558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8153.687886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8153.707675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8153.902853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8154.710899] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8165.370923] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8171.285503] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8171.500377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8177.036588] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8177.308314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8179.687398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.707618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8179.712926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8179.736886] Lustre: Skipped 1 previous similar message [ 8184.823704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8184.836271] Lustre: Skipped 1 previous similar message [ 8185.482897] Lustre: server umount lustre-OST0000 complete [ 8189.922924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.954737] Lustre: Skipped 1 previous similar message [ 8189.963179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.970071] Lustre: Skipped 1 previous similar message [ 8195.049614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8195.059648] Lustre: Skipped 1 previous similar message [ 8195.696283] Lustre: server umount lustre-MDT0000 complete [ 8199.762136] LustreError: 285336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452301 with bad export cookie 10356826101518384292 [ 8199.778081] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8199.782442] LustreError: 285336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8200.304218] Lustre: server umount lustre-MDT0001 complete [ 8209.140630] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:18:29 (1788452309) [ 8215.516343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8216.126077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8221.058403] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8221.158804] LustreError: 289724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8221.182683] LustreError: 289724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8226.277149] LustreError: 289725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8230.166601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8230.473906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8235.638416] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8244.858662] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8252.690403] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8261.759574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8262.098972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8269.628751] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8279.167035] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8301.026067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8301.036543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8301.057921] Lustre: Skipped 1 previous similar message [ 8301.068077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8302.060709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8304.981815] Lustre: server umount lustre-OST0000 complete [ 8312.290472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8312.295769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8312.295920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8312.302477] Lustre: Skipped 2 previous similar messages [ 8312.316725] Lustre: Skipped 3 previous similar messages [ 8314.466357] Lustre: server umount lustre-MDT0000 complete [ 8317.409496] LustreError: 290454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8317.422977] LustreError: 290454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8317.920022] LustreError: 292535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452419 with bad export cookie 10356826101518385174 [ 8317.932519] LustreError: 292535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8317.932933] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8318.173266] Lustre: server umount lustre-MDT0001 complete [ 8329.317975] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 8332.176790] Key type lgssc unregistered [ 8332.587123] LNet: 293291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8332.600688] LNetError: 293291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8332.633073] LNet: Removed LNI 192.168.201.115@tcp [ 8333.684541] Key type .llcrypt unregistered [ 8333.686345] Key type ._llcrypt unregistered [ 8362.961440] Key type ._llcrypt registered [ 8362.964544] Key type .llcrypt registered [ 8363.130270] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 8377.886889] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8378.641414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8378.789822] alg: No test for adler32 (adler32-zlib) [ 8379.832332] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8380.190671] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 8381.912669] Key type lgssc registered [ 8383.334385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8431.000785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8431.030087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8432.419324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8432.476872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8432.557375] Lustre: lustre-MDT0000: new disk, initializing [ 8432.650868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8432.669566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8437.055537] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8448.242750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8448.344598] Lustre: 297119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8448.442131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8448.446305] Lustre: Skipped 1 previous similar message [ 8448.514413] Lustre: lustre-MDT0001: new disk, initializing [ 8448.594065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8448.637190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8448.650866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8453.773684] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8465.086292] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8472.187508] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8479.915961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8480.349363] Lustre: lustre-OST2710: new disk, initializing [ 8480.362549] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8480.379435] Lustre: 299087:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8480.580684] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8486.478260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8486.486938] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8486.657294] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8487.276462] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8496.497582] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8504.045981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8504.261647] Lustre: lustre-OST03e8: new disk, initializing [ 8504.270208] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8504.285347] Lustre: 300152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8504.420447] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8510.559955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8510.576130] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8510.697275] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8511.196584] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8519.353720] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8527.360929] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8527.553815] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8531.870803] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8532.123139] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8537.327391] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8537.595485] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8543.012690] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8543.289739] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8550.653939] Lustre: server umount lustre-MDT0000 complete [ 8550.882026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8550.894936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8551.923927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8555.143504] LustreError: 299085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452656 with bad export cookie 17706229353198452768 [ 8555.150468] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8555.165737] LustreError: 299085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8555.545883] Lustre: server umount lustre-MDT0001 complete [ 8570.372865] Lustre: server umount lustre-OST2710 complete [ 8573.408350] Lustre: 294918:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452658/real 1788452658] req@000000008b67360c x1875328348788224/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8573.453040] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8573.465697] Lustre: Skipped 2 previous similar messages [ 8575.599155] Lustre: server umount lustre-OST03e8 complete [ 8599.396648] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 8610.040737] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8646.579356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8646.827320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8646.850686] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8646.945614] Lustre: lustre-MDT0000: new disk, initializing [ 8647.077824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.100824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8651.398832] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8662.072833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8662.198327] Lustre: 305400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8662.247719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8662.251586] Lustre: Skipped 1 previous similar message [ 8662.319493] Lustre: lustre-MDT0001: new disk, initializing [ 8662.407306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8662.442041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8662.458469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8667.204411] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8676.596822] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8681.801134] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8688.878476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8689.094946] Lustre: lustre-OST0000: new disk, initializing [ 8689.098329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8689.103363] Lustre: 307368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8689.184678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8695.616515] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8698.487921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8698.507192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8698.679049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8705.213996] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8710.525138] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8710.761057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8715.512893] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8715.747846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8719.332064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8719.333619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8719.365802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8719.377631] LustreError: Skipped 1 previous similar message [ 8719.393317] Lustre: Skipped 1 previous similar message [ 8724.454027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8724.467677] Lustre: Skipped 1 previous similar message [ 8729.570354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8729.593619] Lustre: Skipped 1 previous similar message [ 8732.128135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8732.410832] Lustre: server umount lustre-OST0000 complete [ 8739.321734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8739.355466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8739.379200] Lustre: Skipped 1 previous similar message [ 8739.397433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8743.099953] Lustre: server umount lustre-MDT0000 complete [ 8744.963845] LustreError: 306639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8744.996961] LustreError: 306639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 8748.976629] LustreError: 305391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452850 with bad export cookie 17706229353198454441 [ 8748.987900] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8749.570656] Lustre: server umount lustre-MDT0001 complete [ 8762.075554] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:27:41 (1788452861) [ 8764.564542] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8766.106550] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:27:46 (1788452866) [ 8769.912750] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8801.689499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8802.245641] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8802.400283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8802.588594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.044606] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8820.035451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.121936] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8820.149464] Lustre: 311230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8825.774402] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8847.352769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8847.628510] Lustre: Denying initial registration attempt from nid 192.168.201.115@tcp, specified as failover [ 8847.637088] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8847.647749] LustreError: 312685:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8847.655772] LustreError: 312685:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8847.664866] LustreError: 312685:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8847.744673] Lustre: server umount lustre-OST0000 complete [ 8847.749506] LustreError: 312685:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8856.550741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8856.574124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8856.593594] Lustre: Skipped 1 previous similar message [ 8856.605378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8856.608276] Lustre: Skipped 1 previous similar message [ 8867.815696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8867.822961] Lustre: Skipped 5 previous similar messages [ 8869.344714] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8869.672146] Lustre: server umount lustre-MDT0000 complete [ 8872.933623] LustreError: 311242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8872.950102] LustreError: 311242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8874.152067] LustreError: 311221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452975 with bad export cookie 17706229353198455330 [ 8874.164167] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8874.173995] LustreError: 311221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8874.763380] Lustre: server umount lustre-MDT0001 complete [ 8888.003769] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 8890.768697] Key type lgssc unregistered [ 8891.249501] LNet: 313813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8891.269133] LNetError: 313813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8891.301180] LNet: Removed LNI 192.168.201.115@tcp [ 8892.214497] Key type .llcrypt unregistered [ 8892.217290] Key type ._llcrypt unregistered [ 8905.290893] Key type ._llcrypt registered [ 8905.293690] Key type .llcrypt registered [ 8905.412152] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8906.981144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8907.100556] alg: No test for adler32 (adler32-zlib) [ 8908.383989] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8908.824357] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 8910.544181] Key type lgssc registered [ 8911.717476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8962.556884] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 8974.211128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8974.247126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8975.734132] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8975.760987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8975.940615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8981.080787] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 8991.418279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8991.630909] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8991.707181] Lustre: 316598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8992.197466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8997.884731] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9006.376977] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9012.306586] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9032.384308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9032.770673] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9032.994836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9042.005899] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9052.764266] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9057.762647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9057.767877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.788289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9058.279863] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9058.295780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9058.297921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9058.322592] Lustre: Skipped 1 previous similar message [ 9061.639248] Lustre: server umount lustre-OST0000 complete [ 9068.513872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.532197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9073.643874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9073.646228] Lustre: Skipped 1 previous similar message [ 9073.922417] Lustre: server umount lustre-MDT0000 complete [ 9078.943079] LustreError: 316590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453180 with bad export cookie 7844218664439287745 [ 9078.950925] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9078.954305] LustreError: 316590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9079.377877] Lustre: server umount lustre-MDT0001 complete [ 9092.632156] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 9096.265272] Key type lgssc unregistered [ 9096.643738] LNet: 320179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9096.661988] LNetError: 320179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9096.691452] LNet: Removed LNI 192.168.201.115@tcp [ 9097.752498] Key type .llcrypt unregistered [ 9097.754810] Key type ._llcrypt unregistered [ 9124.958732] Key type ._llcrypt registered [ 9124.960985] Key type .llcrypt registered [ 9125.028548] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9126.466754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9126.540258] alg: No test for adler32 (adler32-zlib) [ 9127.584866] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9127.810504] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 9129.473071] Key type lgssc registered [ 9130.655930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9139.814872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9139.833786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9141.401125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9146.101241] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9150.946035] LustreError: 321596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9150.968203] LustreError: 321596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9155.357659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9155.714150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9162.300452] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9171.935793] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9178.592351] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9187.576699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9188.044832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9194.803948] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9202.538514] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9222.124272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9222.135636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9222.152078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9222.160534] Lustre: Skipped 2 previous similar messages [ 9222.180950] Lustre: Skipped 2 previous similar messages [ 9224.692715] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.15@tcp (stopping) [ 9227.235341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9228.292662] Lustre: server umount lustre-MDT0000 complete [ 9229.779896] LustreError: 322525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9232.359350] LustreError: 322526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9232.387566] LustreError: 322526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9232.877415] LustreError: 321578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453334 with bad export cookie 17399469447136891 [ 9232.897575] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9234.899835] LustreError: 321593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9234.920537] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [ 9234.934487] Lustre: Skipped 2 previous similar messages [ 9237.474403] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.307552] Lustre: server umount lustre-MDT0001 complete [ 9246.491896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9254.324329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9258.197346] LustreError: 323430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9258.219614] LustreError: 323430:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9258.277640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9258.350762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9263.167632] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9266.680837] LustreError: 323431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9266.713230] LustreError: 323431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9267.701795] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9271.420394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9271.759908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9276.573507] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9276.938847] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9284.214692] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9285.751974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9291.106794] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9292.692333] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9297.377263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9297.384342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9297.421934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9297.449274] Lustre: Skipped 1 previous similar message [ 9297.468626] Lustre: Skipped 2 previous similar messages [ 9303.561376] Lustre: server umount lustre-OST0000 complete [ 9307.619190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9307.623441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9307.632234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9307.632241] Lustre: Skipped 2 previous similar messages [ 9307.660610] Lustre: Skipped 1 previous similar message [ 9313.941952] Lustre: server umount lustre-MDT0000 complete [ 9317.629139] LustreError: 321577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453418 with bad export cookie 17399469447144633 [ 9317.635928] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9317.642403] LustreError: 321577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9318.175107] Lustre: server umount lustre-MDT0001 complete [ 9329.657125] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 9332.450586] Key type lgssc unregistered [ 9332.826780] LNet: 327894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9332.834581] LNetError: 327894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9333.862877] LNet: Removed LNI 192.168.201.115@tcp [ 9334.842310] Key type .llcrypt unregistered [ 9334.845935] Key type ._llcrypt unregistered [ 9360.279364] Key type ._llcrypt registered [ 9360.281799] Key type .llcrypt registered [ 9360.450086] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9362.213315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9362.294359] alg: No test for adler32 (adler32-zlib) [ 9363.456164] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9363.743799] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 9365.442460] Key type lgssc registered [ 9366.868691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9374.399317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9374.413324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9375.856114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9380.055761] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9386.478662] LustreError: 329311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9386.498832] LustreError: 329311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9387.948231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9388.300484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9392.558532] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9399.874458] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9405.810213] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9408.998244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9409.005085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9409.019794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9411.041148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.042723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9411.060721] Lustre: Skipped 1 previous similar message [ 9413.155793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9413.161436] Lustre: Skipped 1 previous similar message [ 9418.209652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9418.230655] Lustre: Skipped 1 previous similar message [ 9423.328469] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9423.330677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9423.344804] Lustre: Skipped 1 previous similar message [ 9423.520298] Lustre: server umount lustre-MDT0000 complete [ 9428.454205] LustreError: 329292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453529 with bad export cookie 12062958707223881363 [ 9428.482075] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9428.681389] Lustre: server umount lustre-MDT0001 complete [ 9445.738202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9446.140914] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9446.182330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9446.373683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9451.487262] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9463.280335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9463.491751] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9463.594903] Lustre: 331897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9464.096736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9469.584595] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9479.208761] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9487.724767] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9498.486681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9498.801609] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9498.813375] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9498.827933] LustreError: 333782:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9498.833564] LustreError: 333782:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9498.839865] LustreError: 333782:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9498.926983] Lustre: server umount lustre-OST0000 complete [ 9498.930633] LustreError: 333782:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9510.328522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9510.715546] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9510.967426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9517.121108] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9517.215595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9524.994528] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9532.542277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9532.657525] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9532.755291] Lustre: lustre-OST0001: new disk, initializing [ 9532.759353] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9532.770541] Lustre: 335325:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9532.900626] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9539.205847] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9542.866764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9542.877550] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9543.103362] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9547.872274] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9553.395791] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9553.410553] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9553.414074] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9553.432613] LustreError: Skipped 1 previous similar message [ 9558.494277] Lustre: server umount lustre-OST0001 complete [ 9558.502108] LustreError: 334625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9558.509760] LustreError: 334625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9563.617852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9563.626042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.629820] LustreError: 334249:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9563.631020] LustreError: Skipped 1 previous similar message [ 9563.678690] Lustre: Skipped 2 previous similar messages [ 9568.331594] Lustre: server umount lustre-OST0000 complete [ 9573.858501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9573.863315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9573.879587] Lustre: Skipped 1 previous similar message [ 9573.900788] Lustre: Skipped 4 previous similar messages [ 9578.741896] Lustre: server umount lustre-MDT0000 complete [ 9578.993191] LustreError: 335333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9579.018466] LustreError: 335333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9583.552741] LustreError: 332646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453684 with bad export cookie 12062958707223881916 [ 9583.554190] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9583.566173] LustreError: 332646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9584.114571] Lustre: server umount lustre-MDT0001 complete [ 9598.158470] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [ 9602.165957] Key type lgssc unregistered [ 9602.586334] LNet: 337289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9602.592698] LNetError: 337289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9603.624709] LNet: Removed LNI 192.168.201.115@tcp [ 9605.021644] Key type .llcrypt unregistered [ 9605.027352] Key type ._llcrypt unregistered [ 9683.521504] Key type ._llcrypt registered [ 9683.523674] Key type .llcrypt registered [ 9683.704022] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [ 9699.137732] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [ 9700.667466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9700.699890] alg: No test for adler32 (adler32-zlib) [ 9701.851884] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9702.202824] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [ 9703.864303] Key type lgssc registered [ 9705.073474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9741.684744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9741.722350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9743.164291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9743.198306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9743.288469] Lustre: lustre-MDT0000: new disk, initializing [ 9743.409542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9743.448329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9748.418396] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9758.381807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.515881] Lustre: 342225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9758.577338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9758.583508] Lustre: Skipped 1 previous similar message [ 9758.659557] Lustre: lustre-MDT0001: new disk, initializing [ 9758.808373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9758.872099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9758.894600] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9763.243636] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9772.764761] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9778.880959] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9786.327258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9786.675764] Lustre: lustre-OST0000: new disk, initializing [ 9786.679911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9786.685883] Lustre: 344192:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9786.781966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.459729] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9795.709386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9795.729841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9795.875530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9802.997848] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9807.457666] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9807.647637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9811.485675] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9811.713417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9816.545139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9816.548742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.566782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9816.573527] Lustre: Skipped 1 previous similar message [ 9816.586745] Lustre: Skipped 1 previous similar message [ 9819.696827] Lustre: server umount lustre-OST0000 complete [ 9825.250767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9825.254508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9825.284699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.790922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.798530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9829.543590] Lustre: server umount lustre-MDT0000 complete [ 9832.920558] LustreError: 342217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453934 with bad export cookie 13318198824543914680 [ 9832.937975] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9833.289740] Lustre: server umount lustre-MDT0001 complete [ 9843.688635] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:45:43 (1788453943) [ 9852.479927] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:45:52 (1788453952) [ 9860.491205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9860.748924] LustreError: 346948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9860.771844] LustreError: 346948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9860.814969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9864.538555] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9866.209776] LustreError: 346949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9871.333290] LustreError: 346948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9871.959553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9872.245979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9876.109537] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9882.562501] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9887.607398] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9893.882314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9894.258770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.254488] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9909.533864] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9917.299727] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9919.411887] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9921.036452] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9922.476484] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9924.597829] Lustre: Failing over lustre-MDT0000 [ 9925.128165] Lustre: server umount lustre-MDT0000 complete [ 9926.647668] LustreError: 346943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.162363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.190561] Lustre: Skipped 2 previous similar messages [ 9931.752545] LustreError: 346945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9931.797624] LustreError: 346945:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9941.980205] LustreError: 346944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9942.011235] LustreError: 346944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9944.558989] Lustre: 340449:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454029/real 1788454029] req@00000000f5667421 x1875329735003136/t0(0) o400->MGC192.168.201.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788454045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9944.613877] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9948.286518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9953.762331] LustreError: 340446:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cba5a63a x1875329735008128/t0(0) o250->MGC192.168.201.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9953.986561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9954.035931] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9958.362877] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9959.380725] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [ 9959.403545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9959.452184] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9959.509904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9968.987542] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9971.197619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9974.727132] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9976.561728] Lustre: Failing over lustre-MDT0000 [ 9976.856934] Lustre: server umount lustre-MDT0000 complete [ 9978.858544] LustreError: 346944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9978.889544] LustreError: 346944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 9979.876515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9979.883757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.910139] Lustre: Skipped 2 previous similar messages [ 9996.129924] Lustre: 340448:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454081/real 1788454081] req@00000000f9c1ca13 x1875329735024128/t0(0) o400->MGC192.168.201.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788454097 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9996.162077] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9996.458687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10006.919405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10007.011050] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10010.591084] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10012.093511] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10012.159089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10012.161222] Lustre: Skipped 2 previous similar messages [10012.208112] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10012.296753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10025.083978] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10027.543269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10037.734449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10037.740061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10037.747110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10037.747119] Lustre: Skipped 1 previous similar message [10037.809069] Lustre: Skipped 2 previous similar messages [10040.778449] Lustre: server umount lustre-MDT0000 complete [10042.850778] LustreError: 346949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.879088] LustreError: 346949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [10044.993671] LustreError: 346928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454146 with bad export cookie 13318198824543917403 [10045.003061] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10045.009121] LustreError: 346928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10045.376906] Lustre: server umount lustre-MDT0001 complete [10060.610262] Lustre: server umount lustre-OST0000 complete [10076.998419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10077.504191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10082.055845] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10090.458588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10096.364586] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10107.435668] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10116.523644] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10123.886479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10131.753351] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10142.045706] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10148.038451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10158.569229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10158.571184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10158.574160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10158.574170] Lustre: Skipped 2 previous similar messages [10158.598088] Lustre: Skipped 1 previous similar message [10164.202586] Lustre: server umount lustre-OST0000 complete [10172.907778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10172.921239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10172.931797] Lustre: Skipped 3 previous similar messages [10175.671459] Lustre: server umount lustre-MDT0000 complete [10179.049213] LustreError: 353285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10179.071126] LustreError: 353285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10179.464657] LustreError: 355894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454280 with bad export cookie 13318198824543918145 [10179.465143] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10179.472237] LustreError: 355894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10179.990156] Lustre: server umount lustre-MDT0001 complete [10191.575745] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [10194.358303] Key type lgssc unregistered [10194.615817] LNet: 356851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10194.625443] LNetError: 356851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10194.647268] LNet: Removed LNI 192.168.201.115@tcp [10195.687643] Key type .llcrypt unregistered [10195.689833] Key type ._llcrypt unregistered [10225.839759] Key type ._llcrypt registered [10225.841869] Key type .llcrypt registered [10225.922688] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10227.140612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10227.386065] alg: No test for adler32 (adler32-zlib) [10228.533596] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10228.988647] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [10230.800201] Key type lgssc registered [10232.256264] Lustre: Echo OBD driver; http://www.lustre.org/ [10240.711961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10240.733521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10242.218162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10247.149327] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10251.749813] LustreError: 358368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10251.765474] LustreError: 358368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10256.882729] LustreError: 358369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10257.704677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10258.443804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10264.379766] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10272.206025] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10280.269492] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10288.940583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10289.449720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10296.992099] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10304.740781] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10311.993293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10326.000729] Lustre: server umount lustre-MDT0000 complete [10327.521996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.525772] LustreError: 358365:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10327.529936] Lustre: Skipped 1 previous similar message [10327.548505] LustreError: 358365:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10329.405683] LustreError: 358348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454430 with bad export cookie 4266385036711766089 [10329.419952] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10329.817227] Lustre: server umount lustre-MDT0001 complete [10344.363475] Lustre: server umount lustre-OST0000 complete [10355.961455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.456732] LustreError: 362101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10356.565939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10360.639846] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10361.827858] LustreError: 362102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10368.899868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10369.326135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10373.426220] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10380.574135] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10387.264466] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10397.116816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10397.350654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10401.508766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10403.962985] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10411.602352] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10424.291657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10424.296944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10424.304590] Lustre: Skipped 3 previous similar messages [10424.311022] Lustre: Skipped 1 previous similar message [10429.428371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10429.435676] Lustre: Skipped 3 previous similar messages [10429.727239] Lustre: server umount lustre-MDT0000 complete [10433.284118] LustreError: 362081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454534 with bad export cookie 4266385036711767202 [10433.298581] LustreError: 362081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10433.311128] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10433.712903] Lustre: server umount lustre-MDT0001 complete [10449.012783] Lustre: server umount lustre-OST0000 complete [10465.449386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10465.898628] LustreError: 366011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10465.928431] LustreError: 366011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10465.989350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10470.059994] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10478.520739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10483.910536] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10490.733720] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10497.399382] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10504.134920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10504.474416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10504.479780] Lustre: Skipped 1 previous similar message [10511.542853] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10511.706219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10518.392405] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10526.179174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10526.187392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10526.229923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10529.164154] Lustre: server umount lustre-OST0000 complete [10535.395317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10535.403853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10535.416241] Lustre: Skipped 1 previous similar message [10535.422204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10535.426789] Lustre: Skipped 2 previous similar messages [10539.350303] Lustre: server umount lustre-MDT0000 complete [10542.561861] LustreError: 366011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10542.581731] LustreError: 366011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10543.237569] LustreError: 365991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454644 with bad export cookie 4266385036711768224 [10543.244847] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10543.252589] LustreError: 365991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10543.703820] Lustre: server umount lustre-MDT0001 complete [10555.309400] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [10558.771281] Key type lgssc unregistered [10559.029760] LNet: 369546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10559.043510] LNetError: 369546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10559.095138] LNet: Removed LNI 192.168.201.115@tcp [10560.022373] Key type .llcrypt unregistered [10560.024599] Key type ._llcrypt unregistered [10587.768737] Key type ._llcrypt registered [10587.771251] Key type .llcrypt registered [10587.855769] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10589.021564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10589.135039] alg: No test for adler32 (adler32-zlib) [10590.230737] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10590.549330] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [10592.232858] Key type lgssc registered [10593.620959] Lustre: Echo OBD driver; http://www.lustre.org/ [10600.769508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10600.788463] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.794936] LustreError: 370987:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10600.813896] LustreError: 370987:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10600.820210] LustreError: 370987:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10600.827281] LustreError: 370987:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10600.837571] LustreError: 370987:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10608.385225] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10608.395538] LustreError: 371384:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10608.431673] LustreError: 371384:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10608.441981] LustreError: 371384:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10608.451611] LustreError: 371384:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10608.470917] LustreError: 371384:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10628.070535] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [10630.525273] Key type lgssc unregistered [10630.758761] LNet: 372390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10630.763423] LNetError: 372390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10630.774833] LNet: Removed LNI 192.168.201.115@tcp [10631.495421] Key type .llcrypt unregistered [10631.500078] Key type ._llcrypt unregistered [10652.487857] Key type ._llcrypt registered [10652.490679] Key type .llcrypt registered [10652.639568] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [10668.857621] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [10670.110447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10670.130871] alg: No test for adler32 (adler32-zlib) [10671.284889] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10671.495020] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [10673.152215] Key type lgssc registered [10674.437063] Lustre: Echo OBD driver; http://www.lustre.org/ [10713.640095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10713.656883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10714.966992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10714.996720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10715.083466] Lustre: lustre-MDT0000: new disk, initializing [10715.192735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10715.218104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10719.736785] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10729.687646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10729.869745] Lustre: 375550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10729.942613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10729.946994] Lustre: Skipped 1 previous similar message [10730.079995] Lustre: lustre-MDT0001: new disk, initializing [10730.315920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10730.432151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10730.451042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10736.090283] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10746.105776] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10752.275431] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10759.750889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10760.299899] Lustre: lustre-OST0000: new disk, initializing [10760.309708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10760.341196] Lustre: 377524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10760.548906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10766.928167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10766.934580] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10766.938460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10767.100952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10776.815121] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10780.950756] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10781.157699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10786.705595] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10786.979910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10792.933551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10792.933824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10792.950609] LustreError: Skipped 1 previous similar message [10792.955487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10792.970122] Lustre: Skipped 1 previous similar message [10795.438456] Lustre: server umount lustre-OST0000 complete [10802.145954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10802.161748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10802.183485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10802.197404] Lustre: Skipped 1 previous similar message [10803.172699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10803.189676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10803.207417] Lustre: Skipped 1 previous similar message [10805.896631] Lustre: server umount lustre-MDT0000 complete [10810.294084] LustreError: 375541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454911 with bad export cookie 12402697601884294376 [10810.301539] LustreError: 375541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10810.311130] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10810.788681] Lustre: server umount lustre-MDT0001 complete [10822.189297] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10824.083808] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:02:04 (1788454924) [10833.219729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10834.100190] LustreError: 379995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10834.133600] LustreError: 379995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10834.272944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10839.530909] LustreError: 379996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10840.651291] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10844.641517] LustreError: 379995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.767579] LustreError: 379996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.851112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10852.300478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10859.207502] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10868.002747] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10875.871749] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10883.881771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10884.228813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10890.458261] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10898.873076] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10906.934908] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10907.200881] Lustre: lustre-OST0001: new disk, initializing [10907.209657] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10907.233290] Lustre: 382835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10907.425587] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10914.446400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10914.466668] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10914.646914] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10915.198729] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [10927.625204] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10934.824578] Lustre: lustre-OST0001: Not available for connect from 192.168.201.15@tcp (stopping) [10935.298549] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10935.305484] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10935.317088] Lustre: Skipped 1 previous similar message [10938.488525] Lustre: server umount lustre-OST0001 complete [10939.865354] LustreError: 381819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.635879] LustreError: 383155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.680514] LustreError: 383155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10959.331656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10959.341031] LustreError: Skipped 1 previous similar message [10959.348318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10959.357956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10959.361834] Lustre: Skipped 2 previous similar messages [10963.393920] Lustre: server umount lustre-OST0000 complete [10970.083639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10970.130635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10970.147522] Lustre: Skipped 1 previous similar message [10970.176593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10970.184083] Lustre: Skipped 2 previous similar messages [10974.166979] Lustre: server umount lustre-MDT0000 complete [10976.227947] LustreError: 379995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10976.271432] LustreError: 379995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10978.676208] LustreError: 379974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455079 with bad export cookie 12402697601884295258 [10978.680542] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10978.690382] LustreError: 379974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10985.522421] Lustre: server umount lustre-MDT0001 complete [11000.702517] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11004.118450] Key type lgssc unregistered [11004.539675] LNet: 384825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11004.546770] LNetError: 384825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11004.567637] LNet: Removed LNI 192.168.201.115@tcp [11005.722432] Key type .llcrypt unregistered [11005.724140] Key type ._llcrypt unregistered [11065.642624] Key type ._llcrypt registered [11065.647143] Key type .llcrypt registered [11065.824929] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [11067.352831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11067.707936] alg: No test for adler32 (adler32-zlib) [11068.740565] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11068.926019] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [11070.624226] Key type lgssc registered [11072.095884] Lustre: Echo OBD driver; http://www.lustre.org/ [11080.828188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11080.842534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11082.310842] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11082.340463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11082.614603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11088.357605] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11098.793832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11098.947638] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11098.999944] Lustre: 387643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11099.488818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11104.996888] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11112.504744] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11118.547454] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11122.155364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11122.167284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11122.179457] Lustre: Skipped 1 previous similar message [11127.031713] Lustre: server umount lustre-MDT0000 complete [11131.384809] LustreError: 387637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455232 with bad export cookie 2056688691424690221 [11131.392857] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11131.397032] LustreError: 387637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11131.732064] Lustre: server umount lustre-MDT0001 complete [11141.614299] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:07:21 (1788455241) [11149.239404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11149.944874] LustreError: 390157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.962078] LustreError: 390157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11150.009440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11154.812810] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11155.426060] LustreError: 390158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.551772] LustreError: 390157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11165.328177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.657459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11171.303666] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11179.892198] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11185.516199] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11192.711925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11192.853578] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11193.055866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11201.454759] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11212.969024] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11223.427505] Lustre: Permanently deactivating lustre-OST0000 [11223.452164] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11230.253067] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11232.174577] LustreError: 393097:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11232.181318] LustreError: 393097:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11232.185777] LustreError: 393097:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11233.953163] LustreError: 393145:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11233.957480] LustreError: 393145:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11242.136726] Lustre: server umount lustre-OST0000 complete [11245.759138] LustreError: 393395:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11245.766594] LustreError: 393395:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11251.188151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11251.197855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11251.227237] Lustre: Skipped 1 previous similar message [11251.244354] Lustre: Skipped 1 previous similar message [11256.293074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11256.309965] Lustre: Skipped 2 previous similar messages [11258.475779] Lustre: server umount lustre-MDT0000 complete [11261.415985] LustreError: 392226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11261.445413] LustreError: 392226:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11262.463433] LustreError: 390137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455363 with bad export cookie 2056688691424690802 [11262.467568] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11262.472483] LustreError: 390137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11272.846750] Lustre: server umount lustre-MDT0001 complete [11282.994458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11286.916678] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11290.163593] LustreError: 394557:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11296.315453] Lustre: 394896:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.115@tcp [11299.030645] Lustre: server umount MGS complete [11310.909192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11311.517345] LustreError: 395621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.531527] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11311.551803] Lustre: Skipped 1 previous similar message [11311.629204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11317.238822] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11321.825626] LustreError: 395622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.863974] LustreError: 395622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11328.330249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11328.565323] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11328.622238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11335.842576] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11345.344678] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11354.967078] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11357.260560] Lustre: Permanently reactivating lustre-OST0000 [11357.301985] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11357.313830] Lustre: Skipped 2 previous similar messages [11386.336291] LustreError: 397410:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11396.340256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11396.998343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11405.314530] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11413.942715] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11421.668691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11421.668902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11421.670361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11421.720698] LustreError: Skipped 1 previous similar message [11423.333376] Lustre: server umount lustre-OST0000 complete [11430.882875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11430.896206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11430.906721] Lustre: Skipped 1 previous similar message [11430.914211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11430.921578] Lustre: Skipped 1 previous similar message [11434.007601] Lustre: server umount lustre-MDT0000 complete [11437.025732] LustreError: 395621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11437.056934] LustreError: 395621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11438.147807] LustreError: 395603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455539 with bad export cookie 2056688691424691852 [11438.161479] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11438.778765] Lustre: server umount lustre-MDT0001 complete [11454.873622] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11458.310480] Key type lgssc unregistered [11458.643872] LNet: 399993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11458.661223] LNetError: 399993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11458.685241] LNet: Removed LNI 192.168.201.115@tcp [11459.860652] Key type .llcrypt unregistered [11459.862406] Key type ._llcrypt unregistered [11486.905790] Key type ._llcrypt registered [11486.909092] Key type .llcrypt registered [11487.009428] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [11502.927939] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [11504.989168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11505.065902] alg: No test for adler32 (adler32-zlib) [11506.251986] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11506.651628] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [11508.392305] Key type lgssc registered [11509.785821] Lustre: Echo OBD driver; http://www.lustre.org/ [11552.638680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11552.714561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11554.158760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11554.240231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11554.376629] Lustre: lustre-MDT0000: new disk, initializing [11554.536144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11554.553792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11559.099779] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11569.600393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11569.750560] Lustre: 403149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11569.805679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11569.810617] Lustre: Skipped 1 previous similar message [11569.923817] Lustre: lustre-MDT0001: new disk, initializing [11570.058887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11570.129554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11570.158687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11574.571160] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11585.873479] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11591.780706] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11599.516839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11599.926403] Lustre: lustre-OST0000: new disk, initializing [11599.933713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11599.946710] Lustre: 405119:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11600.087494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11605.577187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11605.582444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11605.667083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11606.312108] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11615.989801] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11621.280409] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11621.438730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11626.927737] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11627.174100] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11631.600171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11631.608549] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11631.608940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11631.627110] Lustre: Skipped 1 previous similar message [11631.636800] LustreError: Skipped 1 previous similar message [11635.692725] Lustre: server umount lustre-OST0000 complete [11641.831919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11641.836938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11641.839523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11641.844566] Lustre: Skipped 1 previous similar message [11641.866628] Lustre: Skipped 2 previous similar messages [11646.655340] Lustre: server umount lustre-MDT0000 complete [11650.322228] LustreError: 403143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455751 with bad export cookie 8276347185765006299 [11650.330459] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11650.332930] LustreError: 403143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11650.564671] Lustre: server umount lustre-MDT0001 complete [11658.593156] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11660.662742] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:16:00 (1788455760) [11668.585963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11669.033818] LustreError: 407583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11669.055695] LustreError: 407583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11669.140558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11674.305569] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11674.608960] LustreError: 407584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.691615] LustreError: 407583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11683.820917] LustreError: 407584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11685.027412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11685.537495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11690.161911] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11698.619960] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11705.090261] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11713.010827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11713.276590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11719.267736] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11727.167821] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11729.137923] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11731.532950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11740.131321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11740.153056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11740.173140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11743.353396] Lustre: server umount lustre-OST0000 complete [11751.404215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11751.416408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11751.424748] Lustre: Skipped 2 previous similar messages [11751.457463] Lustre: Skipped 3 previous similar messages [11753.214202] Lustre: server umount lustre-MDT0000 complete [11756.514132] LustreError: 407584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11756.545991] LustreError: 407584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11757.179648] LustreError: 407564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455858 with bad export cookie 8276347185765007181 [11757.188392] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11757.193574] LustreError: 407564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11757.697614] Lustre: server umount lustre-MDT0001 complete [11770.382744] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11773.788208] Key type lgssc unregistered [11774.268127] LNet: 411314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11774.277594] LNetError: 411314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11774.303692] LNet: Removed LNI 192.168.201.115@tcp [11775.280104] Key type .llcrypt unregistered [11775.282779] Key type ._llcrypt unregistered [11800.471716] Key type ._llcrypt registered [11800.475494] Key type .llcrypt registered [11800.583267] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11800.832314] Key type .llcrypt unregistered [11800.834136] Key type ._llcrypt unregistered [11819.375476] Key type ._llcrypt registered [11819.383755] Key type .llcrypt registered [11819.496186] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [11820.783290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11820.890459] alg: No test for adler32 (adler32-zlib) [11821.960343] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11822.265315] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [11823.952387] Key type lgssc registered [11825.150338] Lustre: Echo OBD driver; http://www.lustre.org/ [11834.327483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11834.348848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11836.038941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11841.703551] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11846.626076] LustreError: 413548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11846.656615] LustreError: 413548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11851.083281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11851.619957] LustreError: 413549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11851.645416] LustreError: 413549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11851.667250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11856.870990] LustreError: 413548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11856.889215] LustreError: 413548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11858.165750] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11860.964894] LustreError: 413544:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11860.983719] LustreError: 413544:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11861.005145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11866.093483] LustreError: 413548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.151387] LustreError: 413548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11866.429570] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11873.973754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11874.608490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11879.345878] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [11890.152158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11890.164039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11890.194296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11895.132888] Lustre: server umount lustre-OST0000 complete [11900.386187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11900.394500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11900.404227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11900.413634] Lustre: Skipped 2 previous similar messages [11900.419391] Lustre: Skipped 1 previous similar message [11905.524267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11905.529032] Lustre: Skipped 2 previous similar messages [11905.733798] Lustre: server umount lustre-MDT0000 complete [11909.137565] LustreError: 413530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456010 with bad export cookie 9352757809236941513 [11909.150052] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11909.493669] Lustre: server umount lustre-MDT0001 complete [11920.472625] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [11923.776846] Key type lgssc unregistered [11924.126690] LNet: 416700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11924.137177] LNetError: 416700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11924.150396] LNet: Removed LNI 192.168.201.115@tcp [11924.909335] Key type .llcrypt unregistered [11924.910666] Key type ._llcrypt unregistered [11955.941827] Key type ._llcrypt registered [11955.945427] Key type .llcrypt registered [11956.072217] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [11957.455691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11957.520644] alg: No test for adler32 (adler32-zlib) [11958.968132] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11959.394586] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [11961.192415] Key type lgssc registered [11963.576273] Lustre: Echo OBD driver; http://www.lustre.org/ [11974.277914] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11974.304646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11989.856458] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11994.976354] LustreError: 418114:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.115@tcp: failed processing log, type 4: rc = -110 [12021.664244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12028.953773] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12038.505124] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12046.470248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12047.011372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12047.076927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12051.500189] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12052.457805] LustreError: 418140:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12052.479974] LustreError: 418140:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12056.581194] LustreError: 419153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.670193] LustreError: 419152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.716619] LustreError: 419152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12062.959493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12063.611548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12071.383197] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12084.553267] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12090.952937] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12098.530104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12098.545087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12098.568946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12099.567321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12099.571186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12099.592336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12104.573638] Lustre: server umount lustre-OST0000 complete [12109.798250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12109.800467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12109.807220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12109.807228] Lustre: Skipped 1 previous similar message [12109.837585] Lustre: Skipped 1 previous similar message [12114.918680] LustreError: 419157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12115.139816] Lustre: server umount lustre-MDT0000 complete [12119.279336] LustreError: 418121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456220 with bad export cookie 18023007047497205287 [12119.284917] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12119.289589] LustreError: 418121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12119.728604] Lustre: server umount lustre-MDT0001 complete [12132.140486] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12136.166240] Key type lgssc unregistered [12136.643417] LNet: 421690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12136.651818] LNetError: 421690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12136.670866] LNet: Removed LNI 192.168.201.115@tcp [12137.612591] Key type .llcrypt unregistered [12137.618045] Key type ._llcrypt unregistered [12166.132788] Key type ._llcrypt registered [12166.135335] Key type .llcrypt registered [12166.238748] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12167.588831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12167.676771] alg: No test for adler32 (adler32-zlib) [12168.766507] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12169.039805] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12170.816174] Key type lgssc registered [12172.708755] Lustre: Echo OBD driver; http://www.lustre.org/ [12182.969924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12183.009096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12184.716440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12191.330247] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12195.327788] LustreError: 423106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12195.365453] LustreError: 423106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12200.426390] LustreError: 423107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12203.204882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12203.935792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12209.971700] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12219.112337] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12227.126248] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12235.186953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12235.539508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12241.704939] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12250.747618] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12255.046102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12260.328873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12260.330817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12260.353830] Lustre: Skipped 1 previous similar message [12260.376276] Lustre: Skipped 2 previous similar messages [12262.930201] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.15@tcp (stopping) [12265.445113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12265.459278] Lustre: Skipped 2 previous similar messages [12266.829208] Lustre: server umount lustre-MDT0000 complete [12267.996223] LustreError: 423103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.590212] LustreError: 423087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.15@tcp arrived at 1788456370 with bad export cookie 12864504423906673192 [12269.611992] LustreError: 423087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12270.565682] LustreError: 423106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12270.605593] LustreError: 423106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12271.868298] LustreError: 423088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456373 with bad export cookie 12864504423906672527 [12271.894967] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12274.659848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12274.663497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12274.682705] Lustre: Skipped 1 previous similar message [12274.693063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12275.681473] LustreError: 424047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12275.701340] LustreError: 424047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12278.462566] Lustre: server umount lustre-OST0000 complete [12286.946492] LustreError: 423106:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12286.983118] LustreError: 423106:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12287.869955] Lustre: server umount lustre-MDT0001 complete [12307.874274] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12312.038339] Key type lgssc unregistered [12312.245640] LNet: 426823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12312.258424] LNetError: 426823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12312.301573] LNet: Removed LNI 192.168.201.115@tcp [12313.659881] Key type .llcrypt unregistered [12313.662876] Key type ._llcrypt unregistered [12345.258755] Key type ._llcrypt registered [12345.261432] Key type .llcrypt registered [12345.477189] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12347.588712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12347.783572] alg: No test for adler32 (adler32-zlib) [12349.247992] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12349.573532] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12351.336660] Key type lgssc registered [12353.380787] Lustre: Echo OBD driver; http://www.lustre.org/ [12363.798311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12363.811894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12365.251893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12371.477694] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12375.528384] LustreError: 428242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12375.576644] LustreError: 428242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12380.641746] LustreError: 428243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12382.263405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12382.951511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12388.156391] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12396.904247] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12402.886914] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12409.867394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12410.312722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12417.008510] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12426.114968] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12434.865738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12443.632711] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [12444.641693] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12444.648362] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12444.677332] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12445.154605] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12448.738793] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [12448.748737] Lustre: Skipped 2 previous similar messages [12453.845677] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [12453.852034] Lustre: Skipped 2 previous similar messages [12455.392542] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12455.617327] Lustre: server umount lustre-MDT0001 complete [12460.530332] LustreError: 428237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12460.578927] LustreError: 428237:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12465.646033] LustreError: 428243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12465.659940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12465.670363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12465.716700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12465.725676] Lustre: Skipped 2 previous similar messages [12469.064617] Lustre: server umount lustre-OST0000 complete [12470.755959] LustreError: 428242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12473.689669] Lustre: server umount lustre-MDT0000 complete [12496.261823] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12499.549868] Key type lgssc unregistered [12499.870658] LNet: 431973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12499.887509] LNetError: 431973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12499.903212] LNet: Removed LNI 192.168.201.115@tcp [12501.003888] Key type .llcrypt unregistered [12501.007385] Key type ._llcrypt unregistered [12535.735590] Key type ._llcrypt registered [12535.737333] Key type .llcrypt registered [12535.907837] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [12554.092203] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12555.412992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12555.524606] alg: No test for adler32 (adler32-zlib) [12556.803773] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12557.166327] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12558.960464] Key type lgssc registered [12560.344651] Lustre: Echo OBD driver; http://www.lustre.org/ [12600.961760] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12622.049425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12622.080361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12623.679309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12623.838707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12624.086730] Lustre: lustre-MDT0000: new disk, initializing [12624.374056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12624.446290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12633.045190] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12650.669815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12651.069443] Lustre: lustre-OST0000: new disk, initializing [12651.074865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12651.082146] Lustre: Skipped 1 previous similar message [12651.092212] Lustre: 437013:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12651.188304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12657.808089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12657.825401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12657.870926] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12658.355083] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12668.503548] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12684.306060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12684.442038] Lustre: 435996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12684.524747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12684.598757] Lustre: lustre-MDT0001: new disk, initializing [12684.722936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12684.785957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12684.838154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12690.863518] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12701.224836] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12701.448569] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12706.275804] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12706.482181] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12711.906082] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12711.915167] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12711.920469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12711.957417] Lustre: Skipped 1 previous similar message [12717.011710] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.15@tcp (stopping) [12717.036418] Lustre: Skipped 2 previous similar messages [12718.034865] Lustre: server umount lustre-MDT0001 complete [12727.292041] LustreError: 436006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12727.322515] LustreError: 436006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12732.434741] LustreError: 436007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12732.454898] LustreError: 436007:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12737.517161] LustreError: 436006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12737.596031] LustreError: 436006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12742.625699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12742.628816] LustreError: 438616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12742.659532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12742.675294] Lustre: Skipped 1 previous similar message [12746.602833] Lustre: server umount lustre-OST0000 complete [12747.754045] LustreError: 436006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12766.176218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12766.749550] Lustre: server umount lustre-MDT0000 complete [12781.900334] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12786.288589] Key type lgssc unregistered [12786.712075] LNet: 440769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12786.721838] LNetError: 440769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12786.748166] LNet: Removed LNI 192.168.201.115@tcp [12788.023940] Key type .llcrypt unregistered [12788.025666] Key type ._llcrypt unregistered [12837.532921] Key type ._llcrypt registered [12837.535883] Key type .llcrypt registered [12837.660920] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [12837.977683] Key type .llcrypt unregistered [12837.979927] Key type ._llcrypt unregistered [12854.345127] Key type ._llcrypt registered [12854.347569] Key type .llcrypt registered [12854.480480] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [12855.956297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12855.983373] alg: No test for adler32 (adler32-zlib) [12857.029792] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12857.329392] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [12859.088165] Key type lgssc registered [12860.601459] Lustre: Echo OBD driver; http://www.lustre.org/ [12889.289927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12889.317526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12890.790869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12890.850767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12890.998981] Lustre: lustre-MDT0000: new disk, initializing [12891.132137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12891.162198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12896.018138] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12907.428244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12907.564913] Lustre: 444153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12907.631820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12907.634802] Lustre: Skipped 1 previous similar message [12907.711811] Lustre: lustre-MDT0001: new disk, initializing [12907.814736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12907.848114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12907.861046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12914.223256] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12926.375211] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12932.641632] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12940.521897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12940.942908] Lustre: lustre-OST0000: new disk, initializing [12940.948574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12940.958482] Lustre: 446123:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12941.057092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12947.539538] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [12948.081171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12948.092880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12948.246102] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12959.729472] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12968.933848] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12968.935426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12968.955700] Lustre: Skipped 2 previous similar messages [12968.959413] Lustre: Skipped 1 previous similar message [12972.348866] Lustre: server umount lustre-MDT0000 complete [12976.114831] LustreError: 444144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457077 with bad export cookie 12199988686648042577 [12976.116160] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12976.141904] LustreError: 444144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12976.589921] Lustre: server umount lustre-MDT0001 complete [12991.481284] Lustre: server umount lustre-OST0000 complete [13002.192537] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:38:22 (1788457102) [13010.351754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13010.988366] LustreError: 448152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13011.008778] LustreError: 448152:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13011.075211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13016.323401] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13016.547652] LustreError: 448153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13016.578964] LustreError: 448153:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13021.670172] LustreError: 448152:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13025.181353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13025.545120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13030.073878] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13037.799961] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13044.041642] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13064.429491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13064.728842] Lustre: Found index 0 for lustre-OST0000, updating log [13065.048463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13072.026884] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13080.140756] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13083.540617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13093.857482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13093.866583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13093.871842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13093.871850] Lustre: Skipped 1 previous similar message [13093.910600] Lustre: Skipped 1 previous similar message [13094.953784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13094.975692] Lustre: Skipped 2 previous similar messages [13098.331778] Lustre: server umount lustre-OST0000 complete [13115.859400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13115.865950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13115.883717] Lustre: Skipped 1 previous similar message [13115.893576] Lustre: Skipped 1 previous similar message [13120.544449] LNet: 1 local NIs in recovery (showing 1): 192.168.201.115@tcp [13121.006433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13121.022746] Lustre: Skipped 1 previous similar message [13130.212611] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13130.519540] Lustre: server umount lustre-MDT0000 complete [13132.261677] LustreError: 448148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13132.278965] LustreError: 448148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13134.969885] LustreError: 448132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457236 with bad export cookie 12199988686648043970 [13134.980114] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13134.986922] LustreError: 448132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13135.693741] Lustre: server umount lustre-MDT0001 complete [13146.177883] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:40:45 (1788457245) [13776.969416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13777.521144] LustreError: 454565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13777.558474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13783.020729] LustreError: 454566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13783.182428] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13787.111358] LustreError: 454565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13793.263194] LustreError: 454566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13793.300154] LustreError: 454566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13795.074165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13795.598680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13801.639182] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13810.313315] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13817.861133] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13838.432441] LNet: 1 local NIs in recovery (showing 1): 192.168.201.115@tcp [13842.758295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13843.063707] LustreError: 456403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13843.211160] Lustre: Found index 0 for lustre-OST0000, updating log [13843.446992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13849.242485] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [13857.324455] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13859.810111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13859.820090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13859.836731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13859.839172] Lustre: Skipped 4 previous similar messages [13866.088989] Lustre: server umount lustre-OST0000 complete [13871.012453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13871.030174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13871.038498] Lustre: Skipped 1 previous similar message [13871.059108] Lustre: Skipped 1 previous similar message [13873.125035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13873.138182] Lustre: Skipped 3 previous similar messages [13877.284280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13877.286610] Lustre: Skipped 6 previous similar messages [13884.384308] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13884.778992] Lustre: server umount lustre-MDT0000 complete [13885.487773] LustreError: 454565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13885.510764] LustreError: 454565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13889.521680] LustreError: 454546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788457990 with bad export cookie 12199988686648045055 [13889.533578] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13889.537148] LustreError: 454546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13890.122620] Lustre: server umount lustre-MDT0001 complete [13897.824345] LNet: 1 local NIs in recovery (showing 1): 192.168.201.115@tcp [13921.629561] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [13935.425578] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [13973.927736] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:54:33 (1788458073) [13992.924948] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [13996.269256] Key type lgssc unregistered [13996.647712] LNet: 461185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13996.655045] LNetError: 461185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14104.172621] LNet: Removed LNI 192.168.201.115@tcp [14105.288447] Key type .llcrypt unregistered [14105.291362] Key type ._llcrypt unregistered [14125.085840] Key type ._llcrypt registered [14125.089467] Key type .llcrypt registered [14125.270192] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [14126.865386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14127.009461] alg: No test for adler32 (adler32-zlib) [14128.203141] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14128.508580] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [14130.288234] Key type lgssc registered [14131.361341] Lustre: Echo OBD driver; http://www.lustre.org/ [14135.869611] LNet: Added LNI 192.168.201.115@tcp100 [8/256/0/180] [14136.878167] LNet: Added LNI 192.168.201.115@tcp101 [8/256/0/180] [14138.040637] LNet: Added LNI 192.168.201.115@tcp102 [8/256/0/180] [14139.128761] LNet: Added LNI 192.168.201.115@tcp103 [8/256/0/180] [14140.333260] LNet: Added LNI 192.168.201.115@tcp104 [8/256/0/180] [14141.292840] LNet: Added LNI 192.168.201.115@tcp105 [8/256/0/180] [14142.344129] LNet: Added LNI 192.168.201.115@tcp106 [8/256/0/180] [14143.436582] LNet: Added LNI 192.168.201.115@tcp107 [8/256/0/180] [14144.524484] LNet: Added LNI 192.168.201.115@tcp108 [8/256/0/180] [14145.689685] LNet: Added LNI 192.168.201.115@tcp109 [8/256/0/180] [14146.806025] LNet: Added LNI 192.168.201.115@tcp110 [8/256/0/180] [14147.883057] LNet: Added LNI 192.168.201.115@tcp111 [8/256/0/180] [14148.769509] LNet: Added LNI 192.168.201.115@tcp112 [8/256/0/180] [14149.716226] LNet: Added LNI 192.168.201.115@tcp113 [8/256/0/180] [14150.769974] LNet: Added LNI 192.168.201.115@tcp114 [8/256/0/180] [14151.693722] LNet: Added LNI 192.168.201.115@tcp115 [8/256/0/180] [14152.645665] LNet: Added LNI 192.168.201.115@tcp116 [8/256/0/180] [14153.735386] LNet: Added LNI 192.168.201.115@tcp117 [8/256/0/180] [14154.771794] LNet: Added LNI 192.168.201.115@tcp118 [8/256/0/180] [14155.754323] LNet: Added LNI 192.168.201.115@tcp119 [8/256/0/180] [14156.651831] LNet: Added LNI 192.168.201.115@tcp120 [8/256/0/180] [14157.513579] LNet: Added LNI 192.168.201.115@tcp121 [8/256/0/180] [14158.332426] LNet: Added LNI 192.168.201.115@tcp122 [8/256/0/180] [14159.280975] LNet: Added LNI 192.168.201.115@tcp123 [8/256/0/180] [14160.263799] LNet: Added LNI 192.168.201.115@tcp124 [8/256/0/180] [14161.258882] LNet: Added LNI 192.168.201.115@tcp125 [8/256/0/180] [14162.279334] LNet: Added LNI 192.168.201.115@tcp126 [8/256/0/180] [14163.087387] LNet: Added LNI 192.168.201.115@tcp127 [8/256/0/180] [14163.960666] LNet: Added LNI 192.168.201.115@tcp128 [8/256/0/180] [14164.730293] LNet: Added LNI 192.168.201.115@tcp129 [8/256/0/180] [14165.723765] LNet: Added LNI 192.168.201.115@tcp130 [8/256/0/180] [14166.650424] LNet: Added LNI 192.168.201.115@tcp131 [8/256/0/180] [14167.897952] LNet: Added LNI 192.168.201.115@tcp132 [8/256/0/180] [14168.779758] LNet: Added LNI 192.168.201.115@tcp133 [8/256/0/180] [14169.799202] LNet: Added LNI 192.168.201.115@tcp134 [8/256/0/180] [14170.841630] LNet: Added LNI 192.168.201.115@tcp135 [8/256/0/180] [14178.493512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14178.537867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14179.711579] Lustre: 464218:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14179.791465] LNetError: 464239:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14179.813961] LNetError: Protocol error connecting to 192.168.201.115@tcp120 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14179.830794] LNetError: Skipped 7 previous similar messages [14179.970195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14180.004471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14180.075434] Lustre: lustre-MDT0000: new disk, initializing [14180.166616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14180.177679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14180.770852] LNet: 8 local NIs in recovery (showing 8): 192.168.201.115@tcp127, 192.168.201.115@tcp126, 192.168.201.115@tcp129, 192.168.201.115@tcp130, 192.168.201.115@tcp128, 192.168.201.115@tcp120, 192.168.201.115@tcp131, 192.168.201.115@tcp132 [14180.772299] LNetError: 461676:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14180.842162] LNetError: 461676:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [14180.865032] LNetError: Protocol error connecting to 192.168.201.115@tcp127 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14180.884363] LNetError: Skipped 6 previous similar messages [14181.863212] LNetError: 464245:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14181.899192] LNetError: Protocol error connecting to 192.168.201.115@tcp131 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14183.907064] LNetError: 464240:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14183.924209] LNetError: 464240:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14183.944064] LNetError: Protocol error connecting to 192.168.201.115@tcp131 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14183.966512] LNetError: Skipped 1 previous similar message [14185.615920] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14188.005926] LNetError: 464250:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14188.022986] LNetError: 464250:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14188.042323] LNetError: Protocol error connecting to 192.168.201.115@tcp131 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14188.051886] LNetError: Skipped 1 previous similar message [14197.368686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14197.591186] Lustre: 464236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14197.721871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14197.724979] Lustre: Skipped 1 previous similar message [14197.806764] Lustre: lustre-MDT0001: new disk, initializing [14197.850505] LNetError: 464243:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14197.862100] LNetError: 464243:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14197.873622] Lustre: 465038:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14197.876211] LNetError: Protocol error connecting to 192.168.201.115@tcp133 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14197.876218] LNetError: Skipped 1 previous similar message [14197.904606] Lustre: 465038:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14198.042170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14198.109553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14198.121410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14203.942068] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14204.502945] Lustre: 465320:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14204.512648] Lustre: 465320:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14215.306456] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14216.610144] LNetError: 464240:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14216.627151] LNetError: 464240:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [14216.648329] LNetError: Protocol error connecting to 192.168.201.115@tcp104 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14216.660461] LNetError: Skipped 16 previous similar messages [14221.770120] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14229.467411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14229.877613] Lustre: lustre-OST0000: new disk, initializing [14229.889776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14229.907220] Lustre: 466210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14230.107087] Lustre: 466215:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14230.128424] Lustre: 466215:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14230.193366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14237.260425] Lustre: 466497:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.115@tcp: can't add NID 192.168.201.115@tcp130: rc = -75 [14237.280946] Lustre: 466497:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14237.365974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14237.378887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14237.611678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14238.851855] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14241.248282] LNet: 4 local NIs in recovery (showing 4): 192.168.201.115@tcp133, 192.168.201.115@tcp135, 192.168.201.115@tcp131, 192.168.201.115@tcp127 [14249.381297] LNetError: 464238:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.115 [14249.397214] LNetError: 464238:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 17 previous similar messages [14249.402490] LNetError: Protocol error connecting to 192.168.201.115@tcp134 at host 192.168.201.115:988: is it running a compatible version of Lustre? [14249.431293] LNetError: Skipped 17 previous similar messages [14250.130457] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14263.270503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14263.287787] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14263.294309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14263.326130] Lustre: Skipped 1 previous similar message [14263.339024] Lustre: Skipped 1 previous similar message [14266.434884] Lustre: server umount lustre-OST0000 complete [14273.506316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14273.539352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14273.544693] Lustre: Skipped 1 previous similar message [14277.494986] Lustre: server umount lustre-MDT0000 complete [14282.040971] LustreError: 464226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458383 with bad export cookie 12112889117180647874 [14282.042740] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14282.057843] LustreError: 464226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14282.560575] Lustre: server umount lustre-MDT0001 complete [14297.236692] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [14300.680957] Key type lgssc unregistered [14301.195371] LNet: 468132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14301.205087] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14301.215267] LNet: Removed LNI 192.168.201.115@tcp [14302.240300] LNet: Removed LNI 192.168.201.115@tcp100 [14302.271166] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14302.305394] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14302.328497] LNet: Removed LNI 192.168.201.115@tcp101 [14303.392246] LNet: Removed LNI 192.168.201.115@tcp102 [14303.394276] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14303.415299] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14303.445569] LNet: Removed LNI 192.168.201.115@tcp103 [14304.480291] LNet: Removed LNI 192.168.201.115@tcp104 [14305.505939] LNet: Removed LNI 192.168.201.115@tcp105 [14305.510589] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14305.520568] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14305.535068] LNet: Removed LNI 192.168.201.115@tcp106 [14306.592191] LNet: Removed LNI 192.168.201.115@tcp107 [14307.616444] LNet: Removed LNI 192.168.201.115@tcp108 [14308.705021] LNet: Removed LNI 192.168.201.115@tcp109 [14309.728210] LNet: Removed LNI 192.168.201.115@tcp110 [14309.742136] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14309.747949] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14309.764961] LNet: Removed LNI 192.168.201.115@tcp111 [14310.820501] LNet: Removed LNI 192.168.201.115@tcp112 [14311.840197] LNet: Removed LNI 192.168.201.115@tcp113 [14312.864836] LNet: Removed LNI 192.168.201.115@tcp114 [14312.880069] LNet: Removed LNI 192.168.201.115@tcp115 [14313.888294] LNet: Removed LNI 192.168.201.115@tcp116 [14314.912220] LNet: Removed LNI 192.168.201.115@tcp117 [14315.936162] LNet: Removed LNI 192.168.201.115@tcp118 [14316.960178] LNet: Removed LNI 192.168.201.115@tcp119 [14317.984179] LNet: Removed LNI 192.168.201.115@tcp120 [14317.988321] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14317.995788] LNetError: 468132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [14318.003868] LNet: Removed LNI 192.168.201.115@tcp121 [14319.073196] LNet: Removed LNI 192.168.201.115@tcp122 [14320.096196] LNet: Removed LNI 192.168.201.115@tcp123 [14320.102627] LNet: Removed LNI 192.168.201.115@tcp124 [14321.185451] LNet: Removed LNI 192.168.201.115@tcp125 [14322.208226] LNet: Removed LNI 192.168.201.115@tcp126 [14322.215216] LNet: Removed LNI 192.168.201.115@tcp127 [14323.232371] LNet: Removed LNI 192.168.201.115@tcp128 [14324.259323] LNet: Removed LNI 192.168.201.115@tcp129 [14325.280230] LNet: Removed LNI 192.168.201.115@tcp130 [14325.284296] LNet: Removed LNI 192.168.201.115@tcp131 [14326.304290] LNet: Removed LNI 192.168.201.115@tcp132 [14326.311637] LNet: Removed LNI 192.168.201.115@tcp133 [14327.329524] LNet: Removed LNI 192.168.201.115@tcp134 [14327.348245] LNet: Removed LNI 192.168.201.115@tcp135 [14328.288327] Key type .llcrypt unregistered [14328.290082] Key type ._llcrypt unregistered [14353.710822] Key type ._llcrypt registered [14353.715776] Key type .llcrypt registered [14353.842245] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [14370.545709] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [14371.898966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14372.164091] alg: No test for adler32 (adler32-zlib) [14373.406651] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14373.725859] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [14375.400182] Key type lgssc registered [14377.241265] Lustre: Echo OBD driver; http://www.lustre.org/ [14414.830666] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:01:54 (1788458514) [14436.798838] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [14439.812205] Key type lgssc unregistered [14440.209255] LNet: 472236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14440.219779] LNetError: 472236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14440.238691] LNet: Removed LNI 192.168.201.115@tcp [14441.554837] Key type .llcrypt unregistered [14441.556595] Key type ._llcrypt unregistered [14459.649815] Key type ._llcrypt registered [14459.657295] Key type .llcrypt registered [14459.818953] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [14461.270622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14461.413213] alg: No test for adler32 (adler32-zlib) [14462.700748] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14463.010799] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [14464.744241] Key type lgssc registered [14466.369896] Lustre: Echo OBD driver; http://www.lustre.org/ [14477.195958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14477.244765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14478.658502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14478.705700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14478.846926] Lustre: lustre-MDT0000: new disk, initializing [14479.020437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14479.067835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14484.299395] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14496.154933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14496.280620] Lustre: 473451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14496.386668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14496.390645] Lustre: Skipped 1 previous similar message [14496.518869] Lustre: lustre-MDT0001: new disk, initializing [14496.663676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14496.734571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14496.740941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14501.049768] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14511.630905] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14518.678308] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14526.439226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14526.677928] Lustre: lustre-OST0000: new disk, initializing [14526.681101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14526.685978] Lustre: 475420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14526.778837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14532.208641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14532.215615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14532.384476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14534.341977] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14545.183961] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14549.904299] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing lnet_if_list [14555.219699] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14555.249587] LNet: Added LNI 192.168.201.115@tcp10 [8/256/0/180] [14560.674787] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14560.704955] LNet: Added LNI 192.168.201.115@tcp11 [8/256/0/180] [14565.016859] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14565.061324] LNet: Added LNI 192.168.201.115@tcp12 [8/256/0/180] [14570.469985] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14570.528772] LNet: Added LNI 192.168.201.115@tcp13 [8/256/0/180] [14576.105739] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14576.133472] LNet: Added LNI 192.168.201.115@tcp14 [8/256/0/180] [14581.796845] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14581.824751] LNet: Added LNI 192.168.201.115@tcp15 [8/256/0/180] [14586.318944] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14586.343938] LNet: Added LNI 192.168.201.115@tcp16 [8/256/0/180] [14590.841902] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14590.874979] LNet: Added LNI 192.168.201.115@tcp17 [8/256/0/180] [14596.446183] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14596.500963] LNet: Added LNI 192.168.201.115@tcp18 [8/256/0/180] [14601.710251] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14601.730840] LNet: Added LNI 192.168.201.115@tcp19 [8/256/0/180] [14606.483438] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14606.519053] LNet: Added LNI 192.168.201.115@tcp20 [8/256/0/180] [14612.650734] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14612.683578] LNet: Added LNI 192.168.201.115@tcp21 [8/256/0/180] [14618.648439] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14618.703278] LNet: Added LNI 192.168.201.115@tcp22 [8/256/0/180] [14624.998928] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14625.030801] LNet: Added LNI 192.168.201.115@tcp23 [8/256/0/180] [14631.873267] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14631.912507] LNet: Added LNI 192.168.201.115@tcp24 [8/256/0/180] [14637.701371] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14637.734038] LNet: Added LNI 192.168.201.115@tcp25 [8/256/0/180] [14644.121614] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14644.152808] LNet: Added LNI 192.168.201.115@tcp26 [8/256/0/180] [14649.979739] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14650.004261] LNet: Added LNI 192.168.201.115@tcp27 [8/256/0/180] [14656.296760] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14656.325891] LNet: Added LNI 192.168.201.115@tcp28 [8/256/0/180] [14662.097767] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14662.164035] LNet: Added LNI 192.168.201.115@tcp29 [8/256/0/180] [14667.971895] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14667.999682] LNet: Added LNI 192.168.201.115@tcp30 [8/256/0/180] [14673.636073] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14673.661450] LNet: Added LNI 192.168.201.115@tcp31 [8/256/0/180] [14679.982626] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14680.017865] LNet: Added LNI 192.168.201.115@tcp32 [8/256/0/180] [14687.196968] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14687.221995] LNet: Added LNI 192.168.201.115@tcp33 [8/256/0/180] [14694.441100] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14694.471724] LNet: Added LNI 192.168.201.115@tcp34 [8/256/0/180] [14702.053181] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14702.084649] LNet: Added LNI 192.168.201.115@tcp35 [8/256/0/180] [14709.148368] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14709.174441] LNet: Added LNI 192.168.201.115@tcp36 [8/256/0/180] [14716.270126] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14716.302609] LNet: Added LNI 192.168.201.115@tcp37 [8/256/0/180] [14725.484470] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14725.512401] LNet: Added LNI 192.168.201.115@tcp38 [8/256/0/180] [14733.861986] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14733.895993] LNet: Added LNI 192.168.201.115@tcp39 [8/256/0/180] [14741.567385] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14741.608660] LNet: Added LNI 192.168.201.115@tcp40 [8/256/0/180] [14749.286160] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14749.306710] LNet: Added LNI 192.168.201.115@tcp41 [8/256/0/180] [14759.099950] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14759.167510] LNet: Added LNI 192.168.201.115@tcp42 [8/256/0/180] [14769.093863] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14769.128333] LNet: Added LNI 192.168.201.115@tcp43 [8/256/0/180] [14779.757817] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14779.805907] LNet: Added LNI 192.168.201.115@tcp44 [8/256/0/180] [14792.446156] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14792.487458] LNet: Added LNI 192.168.201.115@tcp45 [8/256/0/180] [14806.729924] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14806.772925] LNet: Added LNI 192.168.201.115@tcp46 [8/256/0/180] [14818.081803] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14818.110066] LNet: Added LNI 192.168.201.115@tcp47 [8/256/0/180] [14828.149827] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14828.186332] LNet: Added LNI 192.168.201.115@tcp48 [8/256/0/180] [14840.099055] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14840.126435] LNet: Added LNI 192.168.201.115@tcp49 [8/256/0/180] [14850.128840] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14850.170829] LNet: Added LNI 192.168.201.115@tcp50 [8/256/0/180] [14870.497433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14870.499892] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14870.507763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14870.545684] Lustre: Skipped 1 previous similar message [14875.623748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14875.637967] Lustre: Skipped 4 previous similar messages [14876.759104] Lustre: server umount lustre-MDT0000 complete [14880.570379] LustreError: 474256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788458981 with bad export cookie 15804156337871170239 [14880.572111] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14880.584580] LustreError: 474256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14880.739932] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14880.761447] Lustre: Skipped 1 previous similar message [14880.917701] Lustre: server umount lustre-MDT0001 complete [14898.685820] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [14908.369848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14925.212503] LustreError: 475414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14925.249526] LustreError: 475414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14925.304535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14926.309981] LustreError: 475415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14930.763336] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14931.453269] LustreError: 475414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14936.547985] LustreError: 475415:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14936.561539] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14940.878171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14941.465230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14946.410851] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [14947.823858] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14953.605853] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14960.311170] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14986.728562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14986.737323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14986.749375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14987.749144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14988.784252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14988.800461] Lustre: Skipped 2 previous similar messages [14990.246768] Lustre: server umount lustre-OST0000 complete [14997.994338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14998.005064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14998.020519] Lustre: Skipped 1 previous similar message [14998.026981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15009.248209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15009.251754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15009.274796] Lustre: Skipped 5 previous similar messages [15009.439763] Lustre: server umount lustre-MDT0000 complete [15013.057611] LustreError: 473445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459114 with bad export cookie 15804156337871172878 [15013.058342] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15013.066043] LustreError: 473445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15013.526603] Lustre: server umount lustre-MDT0001 complete [15023.234583] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [15025.596557] Key type lgssc unregistered [15025.966206] LNet: 502476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15025.975623] LNetError: 502476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15026.976985] LNet: Removed LNI 192.168.201.115@tcp [15026.979795] LNetError: 502476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15026.985623] LNet: Removed LNI 192.168.201.115@tcp10 [15026.995380] LNet: Removed LNI 192.168.201.115@tcp11 [15027.003269] LNet: Removed LNI 192.168.201.115@tcp12 [15027.008573] LNet: Removed LNI 192.168.201.115@tcp13 [15027.016520] LNet: Removed LNI 192.168.201.115@tcp14 [15027.020459] LNet: Removed LNI 192.168.201.115@tcp15 [15027.026723] LNet: Removed LNI 192.168.201.115@tcp16 [15027.033484] LNet: Removed LNI 192.168.201.115@tcp17 [15027.039715] LNet: Removed LNI 192.168.201.115@tcp18 [15027.044685] LNet: Removed LNI 192.168.201.115@tcp19 [15027.047825] LNet: Removed LNI 192.168.201.115@tcp20 [15027.055411] LNet: Removed LNI 192.168.201.115@tcp21 [15027.062784] LNet: Removed LNI 192.168.201.115@tcp22 [15027.069902] LNet: Removed LNI 192.168.201.115@tcp23 [15027.076603] LNet: Removed LNI 192.168.201.115@tcp24 [15027.085847] LNet: Removed LNI 192.168.201.115@tcp25 [15027.091881] LNet: Removed LNI 192.168.201.115@tcp26 [15027.101407] LNet: Removed LNI 192.168.201.115@tcp27 [15027.108553] LNet: Removed LNI 192.168.201.115@tcp28 [15027.121192] LNet: Removed LNI 192.168.201.115@tcp29 [15027.125600] LNet: Removed LNI 192.168.201.115@tcp30 [15027.134901] LNet: Removed LNI 192.168.201.115@tcp31 [15027.139896] LNet: Removed LNI 192.168.201.115@tcp32 [15027.143555] LNet: Removed LNI 192.168.201.115@tcp33 [15027.148653] LNet: Removed LNI 192.168.201.115@tcp34 [15027.153886] LNet: Removed LNI 192.168.201.115@tcp35 [15027.161491] LNet: Removed LNI 192.168.201.115@tcp36 [15027.167872] LNet: Removed LNI 192.168.201.115@tcp37 [15027.173591] LNet: Removed LNI 192.168.201.115@tcp38 [15027.178908] LNet: Removed LNI 192.168.201.115@tcp39 [15027.190462] LNet: Removed LNI 192.168.201.115@tcp40 [15027.192442] LNet: Removed LNI 192.168.201.115@tcp41 [15027.201744] LNet: Removed LNI 192.168.201.115@tcp42 [15027.212418] LNet: Removed LNI 192.168.201.115@tcp43 [15027.219798] LNet: Removed LNI 192.168.201.115@tcp44 [15027.231401] LNet: Removed LNI 192.168.201.115@tcp45 [15027.238893] LNet: Removed LNI 192.168.201.115@tcp46 [15027.243470] LNet: Removed LNI 192.168.201.115@tcp47 [15027.249842] LNet: Removed LNI 192.168.201.115@tcp48 [15027.254904] LNet: Removed LNI 192.168.201.115@tcp49 [15027.274872] LNet: Removed LNI 192.168.201.115@tcp50 [15028.032308] Key type .llcrypt unregistered [15028.034279] Key type ._llcrypt unregistered [15049.909820] Key type ._llcrypt registered [15049.912241] Key type .llcrypt registered [15050.039748] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [15065.065127] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [15066.319613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15066.355068] alg: No test for adler32 (adler32-zlib) [15067.440673] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15067.776613] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [15069.456351] Key type lgssc registered [15070.638440] Lustre: Echo OBD driver; http://www.lustre.org/ [15107.333426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15107.350840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15108.729643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15108.756782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15108.876353] Lustre: lustre-MDT0000: new disk, initializing [15108.998285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15109.047734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15113.148981] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15123.238868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15123.376175] Lustre: 505674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15123.449077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15123.452317] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15123.454967] Lustre: Skipped 1 previous similar message [15123.523326] Lustre: lustre-MDT0001: new disk, initializing [15123.619891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15123.694597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15123.718593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15128.329461] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15138.664412] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15145.353861] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15155.126197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15155.369215] Lustre: *** cfs_fail_loc=90f, val=0*** [15155.440833] Lustre: lustre-OST0000: new disk, initializing [15155.444052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15155.450112] Lustre: 507695:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15155.579146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15163.314424] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15165.049795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15165.073235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15165.133742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15174.289481] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15185.889376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15185.894618] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15185.903792] LustreError: Skipped 1 previous similar message [15185.908308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15185.920277] Lustre: Skipped 1 previous similar message [15187.251993] Lustre: server umount lustre-OST0000 complete [15190.505320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15190.519934] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15190.541025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15190.550738] Lustre: Skipped 1 previous similar message [15196.133744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15196.153580] Lustre: Skipped 3 previous similar messages [15196.896700] Lustre: server umount lustre-MDT0000 complete [15200.779617] LustreError: 506470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459302 with bad export cookie 13006341274844186657 [15200.797423] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15200.798348] LustreError: 506470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15201.217570] Lustre: server umount lustre-MDT0001 complete [15213.162347] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [15215.934549] Key type lgssc unregistered [15216.224322] LNet: 509603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15216.240289] LNetError: 509603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15216.276499] LNet: Removed LNI 192.168.201.115@tcp [15217.333568] Key type .llcrypt unregistered [15217.335382] Key type ._llcrypt unregistered [15237.748911] Key type ._llcrypt registered [15237.752202] Key type .llcrypt registered [15237.882511] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [15239.300775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15239.352108] alg: No test for adler32 (adler32-zlib) [15240.335497] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15240.545872] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [15242.208244] Key type lgssc registered [15243.728527] Lustre: Echo OBD driver; http://www.lustre.org/ [15252.295974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15252.330656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15253.724359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15258.399611] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15264.234021] LustreError: 510874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15264.249786] LustreError: 510874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15267.892203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15268.419648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15273.061960] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15280.139926] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15285.587986] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15293.118928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15293.451348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15299.911689] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15306.029611] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15320.667345] loop0: detected capacity change from 0 to 204800000 [15320.905478] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15320.917680] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15320.953062] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15328.163671] loop0: detected capacity change from 0 to 204800000 [15328.373524] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15328.380939] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15328.388674] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15334.529456] loop0: detected capacity change from 0 to 204800000 [15340.720651] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15340.848779] Lustre: fs15246-OST0000: new disk, initializing [15340.853109] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15340.859322] Lustre: 514471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15340.922765] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15348.207886] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15358.687170] loop1: detected capacity change from 0 to 204800000 [15364.025526] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15364.153059] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15364.204960] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15364.270919] Lustre: fs15246-MDT0000: new disk, initializing [15364.483235] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15364.511540] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15364.631563] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15365.670297] Lustre: 515594:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000486c98ce x1875335541996800/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788459482 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15369.269600] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15373.810604] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15373.821072] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15373.867202] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15395.829900] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15395.851846] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15397.914166] Lustre: server umount fs15246-MDT0000 complete [15400.939506] LustreError: 510870:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15406.074460] LustreError: 510869:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15407.375173] Lustre: server umount fs15246-OST0000 complete [15420.392580] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15420.408679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15420.432687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15421.423230] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15421.436686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15421.441885] Lustre: Skipped 1 previous similar message [15426.535031] LustreError: 515892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15426.556513] LustreError: 515892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15426.675769] Lustre: server umount lustre-OST0000 complete [15431.665493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15431.686386] Lustre: Skipped 1 previous similar message [15431.693606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15431.718957] Lustre: Skipped 1 previous similar message [15436.774851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15436.783421] Lustre: Skipped 1 previous similar message [15437.441843] Lustre: server umount lustre-MDT0000 complete [15441.585705] LustreError: 510855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459542 with bad export cookie 7195363534018941631 [15441.586135] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15441.597303] LustreError: 510855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15442.124359] Lustre: server umount lustre-MDT0001 complete [15456.440257] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [15459.906897] Key type lgssc unregistered [15460.178052] LNet: 518694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15460.189535] LNetError: 518694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15460.212788] LNet: Removed LNI 192.168.201.115@tcp [15461.283378] Key type .llcrypt unregistered [15461.286213] Key type ._llcrypt unregistered [15478.236277] Key type ._llcrypt registered [15478.246382] Key type .llcrypt registered [15478.361233] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [15479.745537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15479.828477] alg: No test for adler32 (adler32-zlib) [15480.872515] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [15481.099385] LNet: Added LNI 192.168.201.115@tcp [8/256/0/180] [15482.784552] Key type lgssc registered [15483.840724] Lustre: Echo OBD driver; http://www.lustre.org/ [15523.028531] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [15533.356605] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [15569.818786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15569.840653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15571.174641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15571.221179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15571.334862] Lustre: lustre-MDT0000: new disk, initializing [15571.426162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15571.443124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15576.331331] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15586.013150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15586.174298] Lustre: 523143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15586.226273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15586.229703] Lustre: Skipped 1 previous similar message [15586.297475] Lustre: lustre-MDT0001: new disk, initializing [15586.399591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15586.445360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15586.458813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15590.501506] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15599.082609] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15603.793482] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15609.824079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15610.081321] Lustre: lustre-OST0000: new disk, initializing [15610.084077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15610.090726] Lustre: 525109:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15610.176621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15616.082858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15616.102185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15616.240898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15616.619449] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_default_debug -1 all [15625.649312] Lustre: DEBUG MARKER: oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15630.054162] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15630.249281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15634.599680] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15634.849530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15636.962379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15636.966355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15636.978189] LustreError: Skipped 1 previous similar message [15636.995651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15637.001930] Lustre: Skipped 1 previous similar message [15637.012917] Lustre: Skipped 1 previous similar message [15642.092465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15642.106851] Lustre: Skipped 1 previous similar message [15647.202521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15647.209748] Lustre: Skipped 1 previous similar message [15651.296162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15651.619131] Lustre: server umount lustre-OST0000 complete [15657.446568] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15657.461655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15657.464228] Lustre: Skipped 1 previous similar message [15657.475181] Lustre: Skipped 1 previous similar message [15661.661144] Lustre: server umount lustre-MDT0000 complete [15665.314701] LustreError: 523135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788459766 with bad export cookie 3504429805563338862 [15665.319037] LustreError: MGC192.168.201.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15665.329560] LustreError: 523135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15665.901233] Lustre: server umount lustre-MDT0001 complete [15691.387489] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing set_hostid [15703.517346] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing load_modules_local [15736.539982] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15467 sec ========= 14:23:56 (1788459836) [15738.325313] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:23:58 (1788459838) === [15742.415528] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:24:02 (1788459842) === [15795.344184] Lustre: DEBUG MARKER: oleg115-server.virtnet: executing unload_modules_local [15797.564615] Key type lgssc unregistered [15797.829650] LNet: 531760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15797.837192] LNetError: 531760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15797.847906] LNet: Removed LNI 192.168.201.115@tcp [15798.436939] Key type .llcrypt unregistered [15798.438806] Key type ._llcrypt unregistered