[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 557854966 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002198] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006311] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007092] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009131] LSM: Security Framework initializing [ 0.010096] Yama: becoming mindful. [ 0.011000] SELinux: Initializing. [ 0.011059] *** VALIDATE selinux *** [ 0.019613] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023473] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026119] *** VALIDATE tmpfs *** [ 0.028046] *** VALIDATE proc *** [ 0.028889] *** VALIDATE cgroup *** [ 0.029006] *** VALIDATE cgroup2 *** [ 0.030228] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031099] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033000] Spectre V2 : User space: Vulnerable [ 0.033008] Speculative Store Bypass: Vulnerable [ 0.036288] debug: unmapping init [mem 0xffffffffa9c59000-0xffffffffa9c60fff] [ 0.038149] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039541] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040027] ... version: 2 [ 0.040916] ... bit width: 48 [ 0.041021] ... generic registers: 4 [ 0.041875] ... value mask: 0000ffffffffffff [ 0.042011] ... max period: 00007fffffffffff [ 0.043008] ... fixed-purpose events: 3 [ 0.043818] ... event mask: 000000070000000f [ 0.044334] rcu: Hierarchical SRCU implementation. [ 0.046410] smp: Bringing up secondary CPUs ... [ 0.047771] x86: Booting SMP configuration: [ 0.048021] .... node #0, CPUs: #1 #2 #3 [ 0.057032] smp: Brought up 1 node, 4 CPUs [ 0.058916] smpboot: Max logical packages: 1 [ 0.059009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.122118] node 0 deferred pages initialised in 60ms [ 0.125558] devtmpfs: initialized [ 0.127377] x86/mm: Memory block size: 128MB [ 0.130664] gcov: version magic: 0x41383552 [ 0.133443] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.137100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139291] pinctrl core: initialized pinctrl subsystem [ 0.140212] [ 0.140478] ************************************************************* [ 0.142012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144008] ** ** [ 0.147014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.150022] ** ** [ 0.152081] ** This means that this kernel is built to expose internal ** [ 0.155015] ** IOMMU data structures, which may compromise security on ** [ 0.157013] ** your system. ** [ 0.160016] ** ** [ 0.163015] ** If you see this message and you are not debugging the ** [ 0.166015] ** kernel, report this immediately to your vendor! ** [ 0.167006] ** ** [ 0.169009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170006] ************************************************************* [ 0.172804] NET: Registered protocol family 16 [ 0.175594] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.178061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.185561] cpuidle: using governor menu [ 0.186736] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188492] PCI: Using configuration type 1 for base access [ 0.190169] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197202] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.200056] cryptd: max_cpu_qlen set to 1000 [ 0.202206] ACPI: Added _OSI(Module Device) [ 0.203009] ACPI: Added _OSI(Processor Device) [ 0.204012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210421] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216153] ACPI: Interpreter enabled [ 0.217046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218019] ACPI: Using IOAPIC for interrupt routing [ 0.219084] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231022] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.233031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.235015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241195] acpiphp: Slot [2] registered [ 0.242076] acpiphp: Slot [5] registered [ 0.243007] acpiphp: Slot [6] registered [ 0.243945] acpiphp: Slot [7] registered [ 0.245085] acpiphp: Slot [8] registered [ 0.245981] acpiphp: Slot [9] registered [ 0.246066] acpiphp: Slot [10] registered [ 0.247046] acpiphp: Slot [3] registered [ 0.248007] acpiphp: Slot [4] registered [ 0.248980] acpiphp: Slot [11] registered [ 0.250052] acpiphp: Slot [12] registered [ 0.251038] acpiphp: Slot [13] registered [ 0.252036] acpiphp: Slot [14] registered [ 0.252987] acpiphp: Slot [15] registered [ 0.254068] acpiphp: Slot [16] registered [ 0.254988] acpiphp: Slot [17] registered [ 0.256054] acpiphp: Slot [18] registered [ 0.257016] acpiphp: Slot [19] registered [ 0.258007] acpiphp: Slot [20] registered [ 0.258947] acpiphp: Slot [21] registered [ 0.260058] acpiphp: Slot [22] registered [ 0.261051] acpiphp: Slot [23] registered [ 0.262042] acpiphp: Slot [24] registered [ 0.262981] acpiphp: Slot [25] registered [ 0.263071] acpiphp: Slot [26] registered [ 0.265240] acpiphp: Slot [27] registered [ 0.268135] acpiphp: Slot [28] registered [ 0.270085] acpiphp: Slot [29] registered [ 0.272109] acpiphp: Slot [30] registered [ 0.273197] acpiphp: Slot [31] registered [ 0.275070] PCI host bridge to bus 0000:00 [ 0.277016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.281023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.284020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.287020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.290026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.294032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.296163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.299443] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.303784] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.313994] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.318623] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.321012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.322011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.326908] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.329808] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.331034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.333943] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.341013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.354058] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.367643] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.374013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.380013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.403034] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.419383] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.431015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.438026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.467020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.478000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.484016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.494022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.512017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.522213] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.531021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.539018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.554023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.563568] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.575016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.580015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.599017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.608000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.615015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.621020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.639016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.652919] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.654292] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.655000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.656215] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.657000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.660009] iommu: Default domain type: Passthrough [ 0.661845] SCSI subsystem initialized [ 0.662000] ACPI: bus type USB registered [ 0.663195] usbcore: registered new interface driver usbfs [ 0.666120] usbcore: registered new interface driver hub [ 0.668193] usbcore: registered new device driver usb [ 0.670316] pps_core: LinuxPPS API ver. 1 registered [ 0.673019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.676093] PTP clock support registered [ 0.679440] EDAC MC: Ver: 3.0.0 [ 0.682188] PCI: Using ACPI for IRQ routing [ 0.684942] NetLabel: Initializing [ 0.687020] NetLabel: domain hash size = 128 [ 0.689010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.692108] NetLabel: unlabeled traffic allowed by default [ 0.695007] vgaarb: loaded [ 0.696719] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.699010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.709658] clocksource: Switched to clocksource kvm-clock [ 0.866421] VFS: Disk quotas dquot_6.6.0 [ 0.886141] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.889063] *** VALIDATE ramfs *** [ 0.890210] *** VALIDATE hugetlbfs *** [ 0.891628] pnp: PnP ACPI init [ 0.893292] pnp: PnP ACPI: found 6 devices [ 0.915080] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.918575] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.920978] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.923015] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.925657] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.928757] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.931715] NET: Registered protocol family 2 [ 0.934613] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.940774] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.943861] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.947947] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.950865] TCP: Hash tables configured (established 65536 bind 65536) [ 0.953246] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.955905] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.957713] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.959846] NET: Registered protocol family 1 [ 0.962922] RPC: Registered named UNIX socket transport module. [ 0.965578] RPC: Registered udp transport module. [ 0.967651] RPC: Registered tcp transport module. [ 0.969439] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.971882] NET: Registered protocol family 44 [ 0.973778] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.976295] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.978974] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.981378] PCI: CLS 0 bytes, default 64 [ 0.983233] Unpacking initramfs... [ 2.536087] debug: unmapping init [mem 0xffff8ccbbcc54000-0xffff8ccbbffbffff] [ 2.540615] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.542498] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.544990] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.286654] Initialise system trusted keyrings [ 3.288995] Key type blacklist registered [ 3.291077] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.303242] zbud: loaded [ 3.307071] *** VALIDATE nfs *** [ 3.308501] *** VALIDATE nfs4 *** [ 3.310254] pstore: using deflate compression [ 3.322155] Platform Keyring initialized [ 3.492326] NET: Registered protocol family 38 [ 3.495795] Key type asymmetric registered [ 3.497596] Asymmetric key parser 'x509' registered [ 3.503617] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.508433] io scheduler mq-deadline registered [ 3.511169] io scheduler kyber registered [ 3.514241] io scheduler bfq registered [ 3.518450] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.525550] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.529599] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.536616] ACPI: Power Button [PWRF] [ 3.545272] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.561896] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.583146] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.594610] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.631350] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.661434] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.690540] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.696325] Non-volatile memory driver v1.3 [ 3.698631] Linux agpgart interface v0.103 [ 3.741219] virtio_blk virtio1: [vda] 146656 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.745032] vda: detected capacity change from 0 to 75087872 [ 3.774653] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.778850] vdb: detected capacity change from 0 to 1073741824 [ 3.795662] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.799568] vdc: detected capacity change from 0 to 2621440000 [ 3.842802] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.846304] vdd: detected capacity change from 0 to 2621440000 [ 3.888756] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.892362] vde: detected capacity change from 0 to 4294967296 [ 3.925602] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.928746] vdf: detected capacity change from 0 to 4294967296 [ 4.079248] libphy: Fixed MDIO Bus: probed [ 4.086546] usbcore: registered new interface driver usbserial_generic [ 4.088795] usbserial: USB Serial support registered for generic [ 4.091337] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.096262] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.098142] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.100913] mousedev: PS/2 mouse device common for all mice [ 4.104107] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.104639] rtc_cmos 00:05: RTC can wake from S4 [ 4.114926] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.119222] rtc_cmos 00:05: registered as rtc0 [ 4.119333] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.119389] intel_pstate: CPU model not supported [ 4.133806] hid: raw HID events driver (C) Jiri Kosina [ 4.141856] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.145792] usbcore: registered new interface driver usbhid [ 4.154800] usbhid: USB HID core driver [ 4.156573] drop_monitor: Initializing network drop monitor service [ 4.158663] Initializing XFRM netlink socket [ 4.160796] NET: Registered protocol family 10 [ 4.165577] Segment Routing with IPv6 [ 4.166868] NET: Registered protocol family 17 [ 4.168697] mpls_gso: MPLS GSO support [ 4.173348] RAS: Correctable Errors collector initialized. [ 4.175674] AVX version of gcm_enc/dec engaged. [ 4.177343] AES CTR mode by8 optimization enabled [ 4.259490] sched_clock: Marking stable (4259398154, 0)->(5123360061, -863961907) [ 4.270415] registered taskstats version 1 [ 4.272218] Loading compiled-in X.509 certificates [ 4.277305] zswap: loaded using pool lzo/zbud [ 4.303687] Key type big_key registered [ 4.319639] Key type encrypted registered [ 4.321695] ima: No TPM chip found, activating TPM-bypass! [ 4.324797] ima: Allocated hash algorithm: sha1 [ 4.327145] ima: No architecture policies found [ 4.329558] evm: Initialising EVM extended attributes: [ 4.331756] evm: security.selinux [ 4.333336] evm: security.ima [ 4.334572] evm: security.capability [ 4.336137] evm: HMAC attrs: 0x1 [ 4.338752] rtc_cmos 00:05: setting system clock to 2026-09-08 01:17:28 UTC (1788830248) [ 4.347975] debug: unmapping init [mem 0xffffffffaac03000-0xffffffffaadfffff] [ 4.352121] debug: unmapping init [mem 0xffffffffa9982000-0xffffffffa9c58fff] [ 4.361079] Write protecting the kernel read-only data: 28672k [ 4.366089] debug: unmapping init [mem 0xffffffffa8003000-0xffffffffa81fffff] [ 4.368620] debug: unmapping init [mem 0xffffffffa8914000-0xffffffffa89fffff] [ 4.413219] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.420814] systemd[1]: Detected virtualization kvm. [ 4.422410] systemd[1]: Detected architecture x86-64. [ 4.424139] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.450812] systemd[1]: No hostname configured. [ 4.452742] systemd[1]: Set hostname to . [ 4.454684] random: systemd: uninitialized urandom read (16 bytes read) [ 4.475698] systemd[1]: Initializing machine ID from random generator. [ 4.606684] random: ln: uninitialized urandom read (6 bytes read) [ 4.749511] random: systemd: uninitialized urandom read (16 bytes read) [ 4.753564] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.758812] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.763604] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.047647] device-mapper: uevent: version 1.0.3 [ 7.052582] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 8.307764] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started udev Coldplug all Devices. [ 8.488967] random: fast init done [ 9.697349] scsi host0: ata_piix [ 9.715807] scsi host1: ata_piix [ 9.720381] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.732931] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 Starting dracut initqueue hook... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 16.972468] random: crng init done [ 16.981863] random: 7 urandom warning(s) missed due to ratelimiting [ 19.631933] dracut-initqueue[571]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.665502] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.526668] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.833503] SELinux: Disabled at runtime. [ 25.029517] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.035940] systemd[1]: Detected virtualization kvm. [ 25.038716] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.096421] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.099898] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.105595] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.357688] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.361749] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.832674] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.850739] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [FAILED[ 28.477494] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 31.410408] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.564930] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.063865] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 36.885972] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 36.987928] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (11s / no limit) [*** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit)[ 41.337299] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (14s / no limit)[ 41.784560] NFS: Registering the id_resolver key type [ 41.787559] Key type id_resolver registered [ 41.789759] Key type id_legacy registered [ *] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (15s / no limit) [ ***] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 46.908910] hrtimer: interrupt took 3910693 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg239-server login: [ 111.359813] libcfs: loading out-of-tree module taints kernel. [ 111.416089] Key type ._llcrypt registered [ 111.419084] Key type .llcrypt registered [ 111.557910] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 127.128890] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 128.460681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.482388] alg: No test for adler32 (adler32-zlib) [ 129.939886] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 130.588539] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 132.296393] Key type lgssc registered [ 133.601040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 149.105346] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 186.685927] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 199.106569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 199.136133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.323745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 200.344820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 200.517734] Lustre: lustre-MDT0000: new disk, initializing [ 200.638080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.673730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 204.734431] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.071538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 217.123731] Lustre: 6495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 217.142443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 217.146233] Lustre: Skipped 1 previous similar message [ 217.215724] Lustre: lustre-MDT0001: new disk, initializing [ 217.267396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.313095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.324867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 221.076682] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.447088] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 233.939201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.161801] Lustre: lustre-OST0000: new disk, initializing [ 234.165910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 234.173037] Lustre: 8432:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.186425] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 234.251299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.590380] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.641265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 239.648286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 239.703345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 251.307872] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 251.422671] Lustre: lustre-OST0001: new disk, initializing [ 251.427474] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 251.435676] Lustre: 9504:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 251.500327] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 256.347846] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.616965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 257.625646] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 257.691390] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 266.640608] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 272.837365] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 278.884918] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing check_logdir /tmp/testlogs/ [ 283.052718] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing yml_node [ 286.626553] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 289.512262] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 292.163858] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 293.644343] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 21:22:16 EDT 2026 [ 308.808949] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 310.590695] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 329.187232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.195186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.198311] Lustre: Skipped 3 previous similar messages [ 329.201295] Lustre: Skipped 2 previous similar messages [ 331.909370] Lustre: server umount lustre-MDT0000 complete [ 335.650241] LustreError: 6488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830579 with bad export cookie 12812244187196261631 [ 335.656434] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.661884] LustreError: 6488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 339.425286] LustreError: 6502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.426575] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.450709] LustreError: 6502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 339.488563] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.491096] Lustre: Skipped 2 previous similar messages [ 340.448126] LustreError: 6503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.481621] LustreError: 6503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 342.077907] Lustre: server umount lustre-MDT0001 complete [ 345.869422] Lustre: server umount lustre-OST0000 complete [ 349.220737] Lustre: server umount lustre-OST0001 complete [ 355.078856] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 364.329805] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 393.559617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.745357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 393.771063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 393.832301] Lustre: lustre-MDT0000: new disk, initializing [ 393.904722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 393.925856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 397.550977] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 407.270602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 407.383828] Lustre: 15057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 407.435494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 407.439294] Lustre: Skipped 1 previous similar message [ 407.498267] Lustre: lustre-MDT0001: new disk, initializing [ 407.574719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 407.599519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 407.615523] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 411.073765] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 419.212736] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 423.412941] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 428.590644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 428.834269] Lustre: lustre-OST0000: new disk, initializing [ 428.837962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 428.843416] Lustre: 17024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 428.918173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.308082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 430.313172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 430.393612] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 433.392638] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 441.437836] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 445.165259] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 445.351174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.839590] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.024285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 451.041527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 451.042881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 451.049198] LustreError: Skipped 1 previous similar message [ 451.055441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 451.057893] Lustre: Skipped 2 previous similar messages [ 451.085257] Lustre: Skipped 3 previous similar messages [ 456.163785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.169956] Lustre: Skipped 1 previous similar message [ 456.822134] Lustre: server umount lustre-OST0000 complete [ 461.286309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.403119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 466.412593] Lustre: Skipped 3 previous similar messages [ 466.567607] Lustre: server umount lustre-MDT0000 complete [ 469.919945] LustreError: 15050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830714 with bad export cookie 12812244187196263164 [ 469.922177] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.928910] LustreError: 15050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 470.209667] Lustre: server umount lustre-MDT0001 complete [ 473.724334] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:25:16 (1788830716) [ 475.892683] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 477.441895] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:25:20 (1788830720) [ 483.342352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.592156] LustreError: 19431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.657735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.329040] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 488.928870] LustreError: 19432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.026627] LustreError: 19431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.607985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.315711] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 506.077531] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 510.806989] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 516.673440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 522.102911] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 529.308570] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 536.550163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 536.556418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.585765] Lustre: Skipped 1 previous similar message [ 536.596595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 538.598824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 540.991888] Lustre: server umount lustre-OST0000 complete [ 547.296719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 547.302271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.307605] Lustre: Skipped 1 previous similar message [ 550.127478] Lustre: server umount lustre-MDT0000 complete [ 552.987297] LustreError: 19414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788830797 with bad export cookie 12812244187196264039 [ 552.993414] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.997235] LustreError: 19414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 553.305849] Lustre: server umount lustre-MDT0001 complete [ 562.849978] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 565.144658] Key type lgssc unregistered [ 565.468170] LNet: 22960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.477038] LNetError: 22960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 565.504424] LNet: Removed LNI 192.168.202.139@tcp [ 566.385190] Key type .llcrypt unregistered [ 566.387478] Key type ._llcrypt unregistered [ 598.429618] Key type ._llcrypt registered [ 598.435976] Key type .llcrypt registered [ 598.655699] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 600.498236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 600.715836] alg: No test for adler32 (adler32-zlib) [ 601.794643] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 602.213128] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 603.993536] Key type lgssc registered [ 605.870632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 618.429574] loop: module loaded [ 618.589871] loop0: detected capacity change from 0 to 204800000 [ 618.902495] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 618.920732] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.358226] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.909340] loop0: detected capacity change from 0 to 204800000 [ 628.081714] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.089313] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 628.108457] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 636.412590] loop0: detected capacity change from 0 to 204800000 [ 643.426147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 643.442985] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.755784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 644.850515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 644.993521] Lustre: lustre-MDT0000: new disk, initializing [ 645.114231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 645.155484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 650.511897] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 660.925201] loop1: detected capacity change from 0 to 204800000 [ 667.362596] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 667.723572] Lustre: lustre-OST07c6: new disk, initializing [ 667.735229] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 667.742025] Lustre: Skipped 1 previous similar message [ 667.752653] Lustre: 26147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 667.856561] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 673.826869] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 676.993631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 677.001859] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 677.035599] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 679.507985] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 679.523427] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 679.531561] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 679.539808] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 679.551659] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 679.563496] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 694.971827] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 694.992739] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 695.007769] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 695.020641] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 695.031357] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 695.036740] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.770102] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 699.787794] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 699.800232] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 699.808028] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.823488] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.844414] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.862176] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 699.881793] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.894713] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 699.909598] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.928337] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.942185] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.214691] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 703.227647] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 703.238853] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 703.253656] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.265549] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.277735] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.286883] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 703.296405] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.310194] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 703.323472] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.337141] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.347659] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.721153] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 707.736653] Lustre: 25133:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 707.746876] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 707.773231] Lustre: 25133:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.791580] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 707.803660] Lustre: 25133:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.813303] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 707.826717] Lustre: 25133:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.838833] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 707.852130] Lustre: 25133:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.870610] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.877531] Lustre: 25133:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 719.078247] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 719.088113] Lustre: 25132:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 719.094677] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 719.098790] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 719.103659] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 719.111458] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 719.117078] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 719.122652] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 719.127571] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 719.132538] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 719.138180] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 719.143505] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 769.005744] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 769.024489] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 773.244376] Lustre: server umount lustre-OST07c6 complete [ 783.623752] Lustre: server umount lustre-MDT0000 complete [ 800.643592] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 803.932374] Key type lgssc unregistered [ 804.211652] LNet: 28700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.220740] LNetError: 28700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.255870] LNet: Removed LNI 192.168.202.139@tcp [ 805.051993] Key type .llcrypt unregistered [ 805.053930] Key type ._llcrypt unregistered [ 823.138873] Key type ._llcrypt registered [ 823.143772] Key type .llcrypt registered [ 823.249683] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 824.359647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 824.440892] alg: No test for adler32 (adler32-zlib) [ 825.458417] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 825.755098] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 827.472463] Key type lgssc registered [ 828.876665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 836.032238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 836.051598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.424889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.137744] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 847.843218] LustreError: 29966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 847.859891] LustreError: 29966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 851.079843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.529387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 856.150340] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 864.507708] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 870.644917] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 877.623356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 878.065439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.201472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 885.318134] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 894.295191] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 914.607694] Lustre: server umount lustre-OST0000 complete [ 922.081746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.086133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 922.094232] Lustre: Skipped 1 previous similar message [ 922.108695] Lustre: Skipped 1 previous similar message [ 927.203962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 927.215661] Lustre: Skipped 1 previous similar message [ 929.431591] Lustre: server umount lustre-MDT0000 complete [ 932.327422] LustreError: 29966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 932.349885] LustreError: 29966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 933.231348] LustreError: 29946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831177 with bad export cookie 7596152537673280080 [ 933.237786] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 933.243827] LustreError: 29946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 943.820826] Lustre: server umount lustre-MDT0001 complete [ 957.904311] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 961.321389] Key type lgssc unregistered [ 961.672872] LNet: 33706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.698231] LNetError: 33706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.725494] LNet: Removed LNI 192.168.202.139@tcp [ 963.855498] Key type .llcrypt unregistered [ 963.861432] Key type ._llcrypt unregistered [ 985.368596] Key type ._llcrypt registered [ 985.377145] Key type .llcrypt registered [ 985.558956] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 986.637671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 986.680456] alg: No test for adler32 (adler32-zlib) [ 987.960746] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 988.267857] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 990.039765] Key type lgssc registered [ 991.400076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 999.896771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 999.939058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.567327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.100624] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1011.681904] LustreError: 34973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.710351] LustreError: 34973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1016.805099] LustreError: 34974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.494293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.161741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1024.816338] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1033.642665] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1039.586702] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1046.712725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1047.184796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.132507] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1062.002625] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1071.822056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1085.412033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1085.427335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1085.467538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1087.458913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1087.460848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1087.465259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1087.492436] Lustre: Skipped 1 previous similar message [ 1090.203964] Lustre: server umount lustre-OST0000 complete [ 1095.648718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1095.659823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.682426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.695924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1097.697098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.647989] Lustre: server umount lustre-MDT0000 complete [ 1102.818046] LustreError: 34974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.577754] LustreError: 34956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831349 with bad export cookie 3639097568183061782 [ 1105.591941] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1105.596814] LustreError: 34956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1106.143924] Lustre: server umount lustre-MDT0001 complete [ 1118.578181] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1121.848476] Key type lgssc unregistered [ 1122.234579] LNet: 38538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.256662] LNetError: 38538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.323400] LNet: Removed LNI 192.168.202.139@tcp [ 1124.287370] Key type .llcrypt unregistered [ 1124.290617] Key type ._llcrypt unregistered [ 1148.934314] Key type ._llcrypt registered [ 1148.936878] Key type .llcrypt registered [ 1149.018768] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1150.115132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1150.269480] alg: No test for adler32 (adler32-zlib) [ 1151.361577] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1151.661875] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1153.351381] Key type lgssc registered [ 1154.891891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1162.924675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1162.936235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1164.389769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1168.998077] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1173.997261] LustreError: 39951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.031040] LustreError: 39951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1179.070289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.111025] LustreError: 39950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.396104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1184.988899] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1193.366692] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1199.979303] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1208.104721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1208.597602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.678861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1216.262966] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1224.748550] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1230.666386] Lustre: server umount lustre-MDT0000 complete [ 1230.815937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1230.824568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.843687] LustreError: 39951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.374798] LustreError: 39931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831478 with bad export cookie 3786760146359083549 [ 1234.388971] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.437798] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.443170] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1235.455147] Lustre: Skipped 1 previous similar message [ 1235.461192] LustreError: 39945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.499354] LustreError: 39945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1237.008707] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.39@tcp (stopping) [ 1239.456994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1240.804455] Lustre: server umount lustre-MDT0001 complete [ 1270.239463] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1270.550474] Lustre: server umount lustre-OST0000 complete [ 1288.533456] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1291.619696] Key type lgssc unregistered [ 1291.876305] LNet: 43810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.886850] LNetError: 43810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.898886] LNet: Removed LNI 192.168.202.139@tcp [ 1292.773950] Key type .llcrypt unregistered [ 1292.782385] Key type ._llcrypt unregistered [ 1320.801086] Key type ._llcrypt registered [ 1320.802787] Key type .llcrypt registered [ 1321.116904] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1322.927130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1323.070933] alg: No test for adler32 (adler32-zlib) [ 1324.276760] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1324.598263] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1326.304931] Key type lgssc registered [ 1327.654725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1336.125898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1336.154612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.788710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.875260] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1347.051795] LustreError: 45225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.071558] LustreError: 45225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1352.163731] LustreError: 45226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.387147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.856641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1357.641156] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1365.646234] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1371.301303] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1379.349543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1379.996421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1382.071084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1388.462962] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1396.228174] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1404.439874] Lustre: lustre-OST0000: Not available for connect from 192.168.202.39@tcp (stopping) [ 1407.971525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1407.986783] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1408.005673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1408.024562] Lustre: Skipped 1 previous similar message [ 1409.413930] Lustre: server umount lustre-OST0000 complete [ 1427.327559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1427.332887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1427.363154] Lustre: Skipped 2 previous similar messages [ 1427.384809] Lustre: Skipped 1 previous similar message [ 1433.570359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.595371] Lustre: Skipped 3 previous similar messages [ 1433.839430] Lustre: server umount lustre-MDT0000 complete [ 1438.688575] LustreError: 45221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.713205] LustreError: 45221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1438.849271] LustreError: 45208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831683 with bad export cookie 17281688038330782694 [ 1438.863189] LustreError: 45208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1438.865980] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1445.310705] Lustre: server umount lustre-MDT0001 complete [ 1456.417122] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1459.824914] Key type lgssc unregistered [ 1460.196851] LNet: 48934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.203581] LNetError: 48934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.219402] LNet: Removed LNI 192.168.202.139@tcp [ 1461.403221] Key type .llcrypt unregistered [ 1461.405759] Key type ._llcrypt unregistered [ 1490.445264] Key type ._llcrypt registered [ 1490.447538] Key type .llcrypt registered [ 1490.592804] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1491.817130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1492.084460] alg: No test for adler32 (adler32-zlib) [ 1493.561337] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1493.863757] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1495.615460] Key type lgssc registered [ 1497.329309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1505.908878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1505.934976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.620536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1513.595306] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1517.030810] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.055769] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1522.153169] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.211685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.557926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1528.619526] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1535.901532] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1542.402990] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1549.555547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1549.927641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.982928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1556.348747] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1563.894694] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1573.522492] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1576.567760] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1588.534792] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0721h43m50s [ 1590.115631] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1592.316095] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.39@tcp (stopping) [ 1593.383369] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.39@tcp (stopping) [ 1595.366333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.373409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1595.382504] Lustre: Skipped 1 previous similar message [ 1595.416516] Lustre: Skipped 2 previous similar messages [ 1598.443629] LustreError: 50341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.639850] Lustre: server umount lustre-MDT0000 complete [ 1602.557176] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1602.595873] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1603.195076] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1608.679393] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1608.698507] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1610.863759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1610.950408] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1611.207544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.272175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1612.290504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1616.163350] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1620.343387] Lustre: DEBUG MARKER: Wait for df (36669) ... [ 1622.155794] Lustre: DEBUG MARKER: done [ 1626.083279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1626.101651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.121567] Lustre: Skipped 1 previous similar message [ 1626.126243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1626.133574] Lustre: Skipped 3 previous similar messages [ 1626.593197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1631.978495] Lustre: server umount lustre-OST0000 complete [ 1637.857140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1637.864423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.875541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1637.877026] Lustre: Skipped 2 previous similar messages [ 1637.887245] Lustre: Skipped 4 previous similar messages [ 1641.817814] Lustre: server umount lustre-MDT0000 complete [ 1645.045473] LustreError: 54315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.069515] LustreError: 54315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1645.492424] LustreError: 50325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788831889 with bad export cookie 17661220016414170576 [ 1645.504124] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1645.906976] Lustre: server umount lustre-MDT0001 complete [ 1655.957558] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1658.459238] Key type lgssc unregistered [ 1658.772171] LNet: 55269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.777989] LNetError: 55269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.823568] LNet: Removed LNI 192.168.202.139@tcp [ 1661.067352] Key type .llcrypt unregistered [ 1661.071417] Key type ._llcrypt unregistered [ 1704.292713] Key type ._llcrypt registered [ 1704.296572] Key type .llcrypt registered [ 1704.413025] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1706.267071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1706.339570] alg: No test for adler32 (adler32-zlib) [ 1707.463974] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1707.823348] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1709.629830] Key type lgssc registered [ 1711.134382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.602878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1719.626192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.890629] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1720.932675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1721.152479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.668628] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1736.070042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1736.175162] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1736.218744] Lustre: 57474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1736.597404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1741.237829] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1749.166755] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1755.667197] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1762.617448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1762.805397] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1763.109260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.905802] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1770.134138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1777.195942] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1783.084291] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0721h47m05s [ 1784.410799] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1791.497877] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1794.171078] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1794.233504] LustreError: 57483:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1814.532634] Lustre: lustre-MDT0000: Client 1882516c-5ed6-4554-93ab-abd885edbcf1 (at 192.168.202.39@tcp) reconnecting [ 1834.337493] LustreError: 57483:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1836.479232] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1838.659242] Lustre: DEBUG MARKER: done [ 1846.755776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1846.765311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.776772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1847.281784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1847.299192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1847.328442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1847.336799] Lustre: Skipped 1 previous similar message [ 1849.312379] Lustre: server umount lustre-OST0000 complete [ 1854.434942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1854.448202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.487460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1858.595375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1858.601354] Lustre: Skipped 4 previous similar messages [ 1859.405253] Lustre: server umount lustre-MDT0000 complete [ 1862.958087] LustreError: 57467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832107 with bad export cookie 2956495039925544022 [ 1862.959281] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1862.969027] LustreError: 57467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1863.293563] Lustre: server umount lustre-MDT0001 complete [ 1877.078500] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 1880.258590] Key type lgssc unregistered [ 1880.534164] LNet: 61801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.542757] LNetError: 61801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.556294] LNet: Removed LNI 192.168.202.139@tcp [ 1881.381392] Key type .llcrypt unregistered [ 1881.383478] Key type ._llcrypt unregistered [ 1924.336346] Key type ._llcrypt registered [ 1924.338294] Key type .llcrypt registered [ 1924.424906] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 1925.838447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.856797] alg: No test for adler32 (adler32-zlib) [ 1926.931673] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 1927.160640] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 1928.856722] Key type lgssc registered [ 1930.181793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1934.332399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1934.359688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.696085] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1935.728114] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1935.898393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.958159] Lustre: Failing over lustre-MDT0000 [ 1937.433894] Lustre: server umount lustre-MDT0000 complete [ 1944.165769] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:49:46 (1788832186) [ 1950.840854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.210786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1955.804589] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1967.182701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.262180] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1967.286665] Lustre: 64595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1967.547317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1972.392811] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 1979.996436] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1985.705427] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1992.882956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1993.027212] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1993.198496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.517952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2000.596452] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2009.423655] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2015.042466] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2023.905903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2023.911577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.934256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.445491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2025.447319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2029.038319] Lustre: server umount lustre-MDT0000 complete [ 2033.405726] LustreError: 64587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832277 with bad export cookie 18305450932592413489 [ 2033.415823] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2033.420506] LustreError: 64587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2033.996230] Lustre: server umount lustre-MDT0001 complete [ 2038.910791] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2044.919585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2051.048191] Lustre: 63516:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788832279/real 1788832279] req@00000000f29d2924 x1875726482581120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788832295 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2051.114389] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2051.135262] Lustre: Skipped 1 previous similar message [ 2054.111836] Lustre: 63519:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788832282/real 1788832282] req@00000000d18e5fe0 x1875726482581248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788832298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2055.302615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2056.223383] Lustre: 63519:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788832284/real 1788832284] req@00000000514a742d x1875726482581376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788832300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2058.677863] LustreError: 66443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.696970] LustreError: 66443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2058.795244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2058.943769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2059.743697] Lustre: 63516:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788832287/real 1788832287] req@00000000a96eaa74 x1875726482581632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788832303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2060.844127] LustreError: 66444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2060.883235] LustreError: 66444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2063.442101] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2064.928882] Lustre: 63516:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788832293/real 1788832293] req@000000003426728f x1875726482581760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788832309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2065.892445] LustreError: 68363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.914265] LustreError: 68363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2071.012596] LustreError: 68363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.049630] LustreError: 68363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2072.749580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.501684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.267785] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2078.694260] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2080.749799] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2085.697944] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2091.830866] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2099.171261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2099.196368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.226857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2099.229856] Lustre: Skipped 2 previous similar messages [ 2099.924576] Lustre: server umount lustre-MDT0000 complete [ 2101.219971] LustreError: 68364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.246374] LustreError: 68364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2103.747587] LustreError: 64586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832347 with bad export cookie 18305450932592415764 [ 2103.758576] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2103.760680] LustreError: 64586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2104.085797] Lustre: server umount lustre-MDT0001 complete [ 2110.046964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2119.485291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.720442] Lustre: 63517:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788832350/real 1788832350] req@0000000002a94513 x1875726482608384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788832366 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2122.776434] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.793858] Lustre: Skipped 2 previous similar messages [ 2127.903828] LustreError: 63515:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002317b369 x1875726482609536/t0(0) o250->MGC192.168.202.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.324435] LustreError: 66443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.342542] LustreError: 66443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2128.422678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.486283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2133.943454] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2141.678311] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2142.797215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.352366] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2148.514828] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2155.942434] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2160.553067] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2168.803889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2168.810695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.812107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2168.846416] Lustre: Skipped 2 previous similar messages [ 2173.428969] Lustre: server umount lustre-MDT0000 complete [ 2173.921234] LustreError: 71720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.944323] LustreError: 71720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2176.899387] LustreError: 64586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832421 with bad export cookie 18305450932592417143 [ 2176.904859] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2176.908609] LustreError: 64586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2177.173708] Lustre: server umount lustre-MDT0001 complete [ 2183.888055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2186.888921] Lustre: DEBUG MARKER: files compared the same [ 2199.073293] Lustre: server umount lustre-OST0000 complete [ 2216.094818] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 2219.001463] Key type lgssc unregistered [ 2219.275896] LNet: 74319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.287769] LNetError: 74319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.308692] LNet: Removed LNI 192.168.202.139@tcp [ 2220.231120] Key type .llcrypt unregistered [ 2220.235560] Key type ._llcrypt unregistered [ 2244.402114] Key type ._llcrypt registered [ 2244.405389] Key type .llcrypt registered [ 2244.522986] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2245.636435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2245.667922] alg: No test for adler32 (adler32-zlib) [ 2246.950615] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 2247.230083] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 2248.895250] Key type lgssc registered [ 2250.120214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2258.002547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2258.029485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.660284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.079782] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug +malloc all [ 2269.152985] LustreError: 75730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.166492] LustreError: 75730:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2272.354870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.743478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2277.065969] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug +malloc all [ 2285.782674] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2291.919368] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2298.154419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2298.501405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2304.640761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2304.965817] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug +malloc all [ 2312.657131] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2325.478473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2325.485386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.501702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2327.532512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2327.535257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.585254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2329.610363] Lustre: server umount lustre-OST0000 complete [ 2334.177221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2334.196801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.218878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.221805] Lustre: Skipped 1 previous similar message [ 2337.761624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.774629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.345577] Lustre: server umount lustre-MDT0000 complete [ 2342.881522] LustreError: 75729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.917980] LustreError: 75729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2343.019483] LustreError: 75710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832587 with bad export cookie 12741465559667078197 [ 2343.033860] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.293670] Lustre: server umount lustre-MDT0001 complete [ 2354.736845] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 2357.440870] Key type lgssc unregistered [ 2357.821711] LNet: 79259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.836087] LNetError: 79259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.866851] LNet: Removed LNI 192.168.202.139@tcp [ 2358.706474] Key type .llcrypt unregistered [ 2358.714938] Key type ._llcrypt unregistered [ 2383.783062] Key type ._llcrypt registered [ 2383.785306] Key type .llcrypt registered [ 2383.921512] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2385.379908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2385.464718] alg: No test for adler32 (adler32-zlib) [ 2386.577250] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 2386.869060] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 2388.559153] Key type lgssc registered [ 2389.614384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2397.315583] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2397.344095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2412.832067] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2417.955236] LustreError: 80648:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.139@tcp: failed processing log, type 4: rc = -110 [ 2444.639231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2453.366664] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2465.626565] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2474.807964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.938577] Lustre: *** cfs_fail_loc=706, val=0*** [ 2475.329174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.409778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2480.854114] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2484.707951] LustreError: 80673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.723147] LustreError: 80673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2489.833346] LustreError: 81734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.339267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.724258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2495.461960] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2502.828477] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2508.697993] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2511.328233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2511.338955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.340860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2511.359629] Lustre: Skipped 1 previous similar message [ 2516.453680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2516.465258] Lustre: Skipped 2 previous similar messages [ 2521.574983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2525.151161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2525.494772] Lustre: server umount lustre-OST0000 complete [ 2531.810743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.833614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2531.841885] Lustre: Skipped 2 previous similar messages [ 2536.928950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2536.943497] Lustre: Skipped 1 previous similar message [ 2544.097063] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2544.427772] Lustre: server umount lustre-MDT0000 complete [ 2547.168386] LustreError: 81739:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2547.196964] LustreError: 81739:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2549.098697] LustreError: 82474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788832793 with bad export cookie 15527455520317648414 [ 2549.106041] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2549.126655] LustreError: 82474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2549.725979] Lustre: server umount lustre-MDT0001 complete [ 2561.973494] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 2565.334204] Key type lgssc unregistered [ 2565.665212] LNet: 84265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.676422] LNetError: 84265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2566.694289] LNet: Removed LNI 192.168.202.139@tcp [ 2567.635762] Key type .llcrypt unregistered [ 2567.637648] Key type ._llcrypt unregistered [ 2594.144091] Key type ._llcrypt registered [ 2594.147325] Key type .llcrypt registered [ 2594.266789] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2595.784539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2595.836599] alg: No test for adler32 (adler32-zlib) [ 2597.014510] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 2597.317978] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 2598.991265] Key type lgssc registered [ 2600.158587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2608.081236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2608.117797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.164519] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2625.131381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.828741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2630.970061] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2631.135950] LustreError: 86381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.148094] LustreError: 86381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2635.235104] LustreError: 86380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.273223] LustreError: 86380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2639.904560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2640.204591] LustreError: 86379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.238159] LustreError: 86379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2640.260273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2645.480561] LustreError: 86384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.496674] LustreError: 86384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2647.509055] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2650.601806] LustreError: 86385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.487598] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2657.496947] Lustre: MGS: Client 7279f8f2-312b-4b59-9ee9-73c714d444f8 (at 0@lo) reconnecting [ 2657.539073] LustreError: 86395:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2657.894752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.025602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2662.200469] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2678.241080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2678.245575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2678.256712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.268738] Lustre: Skipped 1 previous similar message [ 2681.334036] Lustre: server umount lustre-OST0000 complete [ 2688.485593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.493923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2688.495623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2688.515104] Lustre: Skipped 2 previous similar messages [ 2691.944229] Lustre: server umount lustre-MDT0000 complete [ 2693.602996] LustreError: 87927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.616927] LustreError: 87927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2696.155299] Lustre: server umount lustre-MDT0001 complete [ 2699.954974] Lustre: server umount MGS complete [ 2713.883566] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 22:02:35 (1788832955) [ 2746.539045] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 2758.396862] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2797.641488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2804.054928] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2816.311453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2816.632516] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2816.701458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2816.823186] Lustre: lustre-MDT0001: new disk, initializing [ 2816.937464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2816.985308] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2817.000262] Lustre: 93556:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2817.016769] LustreError: 93556:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2822.408055] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2833.782071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2834.152362] Lustre: lustre-OST0000: new disk, initializing [ 2834.157942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2834.173663] Lustre: 94322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2834.262596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2841.375317] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2845.294901] Lustre: 94779:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000050c1304a x1875727185125632/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788833105 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2851.716878] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2860.195913] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2860.204974] Lustre: MGS: Client d0c60d69-36c5-45e9-9bd8-02c39e1b61e6 (at 0@lo) reconnecting [ 2860.252441] Lustre: 92822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2860.280936] LustreError: 92822:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2860.307952] LustreError: 92822:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2860.362894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2860.624203] Lustre: lustre-MDT0000: new disk, initializing [ 2860.943589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2860.968806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2860.999116] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2862.051963] Lustre: 95391:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000070f89ca1 x1875727185132672/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788833122 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2867.050050] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 2867.346778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2867.359037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2867.407848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2871.292547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2871.309533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2893.282590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2893.283330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.302535] LustreError: Skipped 1 previous similar message [ 2893.306034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.332480] Lustre: Skipped 2 previous similar messages [ 2898.407154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2898.420102] Lustre: Skipped 2 previous similar messages [ 2903.523132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2903.534943] Lustre: Skipped 1 previous similar message [ 2907.103512] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2907.694895] Lustre: server umount lustre-OST0000 complete [ 2913.772124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2913.775627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.789807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2913.812400] Lustre: Skipped 1 previous similar message [ 2919.139652] Lustre: server umount lustre-MDT0000 complete [ 2924.006736] LustreError: 93531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2924.020619] LustreError: 93531:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2930.604984] Lustre: server umount lustre-MDT0001 complete [ 2934.273656] Lustre: server umount MGS complete [ 2945.782803] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 22:06:28 (1788833188) [ 2966.331682] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 2970.432700] Key type lgssc unregistered [ 2970.807783] LNet: 97892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2970.827989] LNetError: 97892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2970.844975] LNet: Removed LNI 192.168.202.139@tcp [ 2972.350524] Key type .llcrypt unregistered [ 2972.353090] Key type ._llcrypt unregistered [ 2998.057279] Key type ._llcrypt registered [ 2998.059101] Key type .llcrypt registered [ 2998.145737] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 2998.791695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2998.837569] alg: No test for adler32 (adler32-zlib) [ 2999.850656] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 3000.244429] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3002.039456] Key type lgssc registered [ 3003.976779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3012.522484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3012.565896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.155353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3019.769925] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3024.354996] LustreError: 99112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.370965] LustreError: 99112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3028.900117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.487634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3034.306689] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3042.622117] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3049.879700] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3059.119871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3059.684981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3068.330196] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3078.338192] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3084.291600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3099.627654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3099.628747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.669563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3102.241989] Lustre: server umount lustre-OST0000 complete [ 3106.274376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3106.285860] LustreError: Skipped 1 previous similar message [ 3106.296975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3106.319642] Lustre: Skipped 1 previous similar message [ 3106.325956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3106.328807] Lustre: Skipped 1 previous similar message [ 3109.858084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3109.861933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3112.435562] Lustre: server umount lustre-MDT0000 complete [ 3114.985137] LustreError: 101155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.002858] LustreError: 101155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3116.156844] LustreError: 99093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833360 with bad export cookie 11269251913390608951 [ 3116.170156] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3116.656161] Lustre: server umount lustre-MDT0001 complete [ 3128.106604] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3131.807718] Key type lgssc unregistered [ 3132.142952] LNet: 102665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3132.155790] LNetError: 102665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3132.175866] LNet: Removed LNI 192.168.202.139@tcp [ 3133.194211] Key type .llcrypt unregistered [ 3133.196095] Key type ._llcrypt unregistered [ 3148.634769] Key type ._llcrypt registered [ 3148.637438] Key type .llcrypt registered [ 3148.806630] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3150.265051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3150.444125] alg: No test for adler32 (adler32-zlib) [ 3151.481702] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 3151.744376] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3153.442741] Key type lgssc registered [ 3154.668255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3161.670899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3161.679128] LustreError: 103742:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3161.694658] LustreError: 103746:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3161.699969] LustreError: 103742:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3161.706743] LustreError: 103742:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3161.713180] LustreError: 103742:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3161.738139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.219272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.744853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.048266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3175.791664] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3181.630395] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing lsmod [ 3184.234884] LustreError: 104946:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3184.272087] LustreError: 104953:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3184.276251] LustreError: 104946:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4961 [ 3184.282613] LustreError: 104946:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3184.290060] LustreError: 104946:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3184.295151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3184.633818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.735685] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.739685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.755632] Lustre: Skipped 2 previous similar messages [ 3189.772435] Lustre: Skipped 2 previous similar messages [ 3193.653213] Lustre: server umount lustre-MDT0000 complete [ 3194.862488] LustreError: 103778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.890902] LustreError: 103778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3195.880884] LustreError: 103779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.910491] LustreError: 103779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3197.430055] LustreError: 103760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833441 with bad export cookie 5868500164287364426 [ 3197.449877] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3197.450903] LustreError: 103760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3197.943318] LustreError: 104214:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c0a59d78 x1875727766016768/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3198.129405] Lustre: server umount lustre-MDT0001 complete [ 3211.222033] Lustre: server umount lustre-OST0000 complete [ 3220.608762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.211615] LustreError: 106017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.321619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.458349] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3226.598727] LustreError: 106018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.717679] LustreError: 106017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3234.804972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.077899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3240.553521] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3249.508217] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3255.704248] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3262.635635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3262.986718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.046756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3270.868903] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3278.460905] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3290.592768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3290.601824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.605957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3290.612389] Lustre: Skipped 1 previous similar message [ 3290.624928] Lustre: Skipped 1 previous similar message [ 3295.423649] Lustre: server umount lustre-OST0000 complete [ 3300.833156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.853029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3300.863032] Lustre: Skipped 1 previous similar message [ 3305.470785] Lustre: server umount lustre-MDT0000 complete [ 3305.953269] LustreError: 106017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3305.973565] LustreError: 106017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3310.514538] LustreError: 106000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833554 with bad export cookie 5868500164287365140 [ 3310.523122] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3310.529129] LustreError: 106000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3311.043689] Lustre: server umount lustre-MDT0001 complete [ 3322.271396] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3324.661738] Key type lgssc unregistered [ 3325.163687] LNet: 109551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3325.178087] LNetError: 109551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3326.259830] LNet: Removed LNI 192.168.202.139@tcp [ 3327.695377] Key type .llcrypt unregistered [ 3327.701866] Key type ._llcrypt unregistered [ 3357.904721] Key type ._llcrypt registered [ 3357.908633] Key type .llcrypt registered [ 3358.029636] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3359.522580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3359.590501] alg: No test for adler32 (adler32-zlib) [ 3360.645887] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 3360.871957] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3362.527375] Key type lgssc registered [ 3364.036764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3374.945692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3374.962521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.605040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.861534] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3386.851224] LustreError: 110969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.865950] LustreError: 110969:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3391.969325] LustreError: 110970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.811702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3393.537754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3400.173078] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3408.473705] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3414.664625] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3421.641097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3421.970623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.023783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3426.806500] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3434.465245] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3444.172726] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3447.233466] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3453.925514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3453.929356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.933661] LustreError: Skipped 1 previous similar message [ 3453.934551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3453.962353] Lustre: Skipped 1 previous similar message [ 3458.602124] Lustre: server umount lustre-OST0000 complete [ 3464.162988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3464.167187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3464.190441] Lustre: Skipped 1 previous similar message [ 3464.207787] Lustre: Skipped 2 previous similar messages [ 3469.281947] LustreError: 110970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3469.296243] LustreError: 110970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3469.594734] Lustre: server umount lustre-MDT0000 complete [ 3473.278903] LustreError: 110951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833717 with bad export cookie 11746127010801944278 [ 3473.280639] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3473.287623] LustreError: 110951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3473.666378] Lustre: server umount lustre-MDT0001 complete [ 3483.268773] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3486.010250] Key type lgssc unregistered [ 3486.383934] LNet: 114615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3486.391929] LNetError: 114615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3487.460901] LNet: Removed LNI 192.168.202.139@tcp [ 3488.421659] Key type .llcrypt unregistered [ 3488.426496] Key type ._llcrypt unregistered [ 3506.880404] Key type ._llcrypt registered [ 3506.883437] Key type .llcrypt registered [ 3506.970178] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3507.840257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3507.879414] alg: No test for adler32 (adler32-zlib) [ 3509.180928] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 3509.459125] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3511.143203] Key type lgssc registered [ 3512.415535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3520.010743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3520.044636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3521.565964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.283191] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3531.747650] LustreError: 115833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3531.768136] LustreError: 115833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3535.690126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.987290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3540.842300] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3549.202489] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3554.801828] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3561.672890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3562.069430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3566.197207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3568.289737] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3576.259932] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3589.093783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3589.100226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.111071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3592.165751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3592.167526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3592.164633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3595.033707] Lustre: server umount lustre-OST0000 complete [ 3602.403260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.409088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3602.428227] Lustre: Skipped 1 previous similar message [ 3602.434927] Lustre: Skipped 1 previous similar message [ 3602.456154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3605.592793] Lustre: server umount lustre-MDT0000 complete [ 3607.529525] LustreError: 115833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3607.573171] LustreError: 115833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3609.474867] LustreError: 115813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788833853 with bad export cookie 1286108290507948918 [ 3609.488209] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3609.498996] LustreError: 115813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3609.889229] Lustre: server umount lustre-MDT0001 complete [ 3620.649183] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3623.897814] Key type lgssc unregistered [ 3624.244276] LNet: 119379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3624.270754] LNetError: 119379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3624.296425] LNet: Removed LNI 192.168.202.139@tcp [ 3625.290141] Key type .llcrypt unregistered [ 3625.299846] Key type ._llcrypt unregistered [ 3650.891133] Key type ._llcrypt registered [ 3650.898735] Key type .llcrypt registered [ 3651.051313] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3652.266404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3652.415159] alg: No test for adler32 (adler32-zlib) [ 3653.611885] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 3653.831690] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3655.511223] Key type lgssc registered [ 3656.613703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3664.890556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3664.916327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3666.381721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3671.160951] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3676.640965] LustreError: 120796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.669921] LustreError: 120796:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3677.702075] LustreError: 120795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.361439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.707465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3686.229802] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3694.547168] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3701.338737] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3708.551529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3708.853726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.952652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3715.681796] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3724.175445] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3733.113546] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3737.506787] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3737.510300] Lustre: Skipped 2 previous similar messages [ 3742.110154] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3742.118896] Lustre: Skipped 2 previous similar messages [ 3742.841690] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3742.875622] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3747.173986] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.39@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3747.183050] Lustre: Skipped 2 previous similar messages [ 3750.943432] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.39@tcp 192.168.0.[2,10]@tcp [ 3755.226755] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.39/24@tcp in log lustre-MDT0000 [ 3755.236815] Lustre: Skipped 2 previous similar messages [ 3757.191207] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.39/24@tcp [ 3757.210316] Lustre: Skipped 1 previous similar message [ 3765.958783] Lustre: server umount lustre-OST0000 complete [ 3771.360853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.367033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3771.371877] Lustre: Skipped 1 previous similar message [ 3776.482972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3776.489692] Lustre: Skipped 2 previous similar messages [ 3780.707479] Lustre: server umount lustre-MDT0000 complete [ 3781.603675] LustreError: 120795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3781.635183] LustreError: 120795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3782.628120] Lustre: 120104:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834010/real 1788834010] req@0000000074ee9399 x1875728292438528/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788834026 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3782.657654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.715524] LustreError: 123427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834028 with bad export cookie 14531906523213562151 [ 3784.721155] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3784.734326] LustreError: 123427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3785.250388] Lustre: server umount lustre-MDT0001 complete [ 3799.293934] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3802.550460] Key type lgssc unregistered [ 3802.910382] LNet: 125491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3802.920865] LNetError: 125491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3802.949228] LNet: Removed LNI 192.168.202.139@tcp [ 3803.999437] Key type .llcrypt unregistered [ 3804.002466] Key type ._llcrypt unregistered [ 3834.685551] Key type ._llcrypt registered [ 3834.688794] Key type .llcrypt registered [ 3834.792402] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 3836.103394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3836.153144] alg: No test for adler32 (adler32-zlib) [ 3837.357364] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 3837.610359] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 3839.295177] Key type lgssc registered [ 3840.380174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3849.352121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3849.386782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3850.875325] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3850.909455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3850.923607] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.39/24@tcp [ 3855.536531] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3861.476796] LustreError: 127050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3861.499712] LustreError: 127050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3864.062804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3864.360944] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3864.388192] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.39/24@tcp [ 3864.448586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3870.153322] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3878.673330] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3886.447673] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3894.630378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3895.302375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.335313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3903.643201] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 3911.396749] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3927.011934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3927.028778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3927.067390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3930.087257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3930.088073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3930.116073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.533581] Lustre: server umount lustre-OST0000 complete [ 3940.321764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3940.359385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3940.364112] Lustre: Skipped 2 previous similar messages [ 3942.761311] Lustre: server umount lustre-MDT0000 complete [ 3945.453552] LustreError: 127050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.467578] LustreError: 127050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3946.444179] LustreError: 127791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834190 with bad export cookie 323140965640294377 [ 3946.475960] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3946.951309] Lustre: server umount lustre-MDT0001 complete [ 3958.475743] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 3961.846951] Key type lgssc unregistered [ 3962.169638] LNet: 130641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3962.184784] LNetError: 130641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3963.242056] LNet: Removed LNI 192.168.202.139@tcp [ 3964.347384] Key type .llcrypt unregistered [ 3964.349064] Key type ._llcrypt unregistered [ 3991.027947] Key type ._llcrypt registered [ 3991.030867] Key type .llcrypt registered [ 3991.161431] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 4009.494940] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 4011.267537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4011.332596] alg: No test for adler32 (adler32-zlib) [ 4012.533922] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 4012.815286] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 4014.607358] Key type lgssc registered [ 4016.096122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4052.878159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4052.921044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4054.190942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4054.212886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4054.303290] Lustre: lustre-MDT0000: new disk, initializing [ 4054.441834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.461974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4059.497876] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4072.436524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4072.609242] Lustre: 134036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4072.684493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4072.687879] Lustre: Skipped 1 previous similar message [ 4072.781592] Lustre: lustre-MDT0001: new disk, initializing [ 4072.890675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4072.932594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4072.945366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4078.924611] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4092.131514] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4098.960841] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4107.072623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.310753] Lustre: lustre-OST0000: new disk, initializing [ 4107.314574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4107.319640] Lustre: 136006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4107.401814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4113.425538] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4114.570472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4114.590276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4114.741981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4122.850735] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4127.961889] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4128.197804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4133.487640] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4133.729918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4140.523323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4140.525164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.538695] LustreError: Skipped 1 previous similar message [ 4140.539719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4140.571741] Lustre: Skipped 1 previous similar message [ 4141.870542] Lustre: server umount lustre-OST0000 complete [ 4149.729646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4149.740113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.777750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4149.780782] Lustre: Skipped 1 previous similar message [ 4150.759043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.113463] Lustre: server umount lustre-MDT0000 complete [ 4155.905379] LustreError: 134030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834400 with bad export cookie 4999779637140939561 [ 4155.913582] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4155.932560] LustreError: 134030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4156.280532] Lustre: server umount lustre-MDT0001 complete [ 4167.123369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.609230] LustreError: 138223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4167.623308] LustreError: 138223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4167.688751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4172.779024] LustreError: 138224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.333183] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4176.869341] LustreError: 138223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.999295] LustreError: 138223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.245657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.730594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4187.382987] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4195.399940] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4202.778978] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4209.470649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4210.030979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4219.150460] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4226.281370] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4231.297483] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4231.498700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4236.134589] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4236.330119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4246.674814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4246.909684] Lustre: lustre-OST0001: new disk, initializing [ 4246.915936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4246.929714] Lustre: 141595:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4247.077275] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4253.805120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4253.817465] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4253.994408] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4254.848437] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4263.930333] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4264.233995] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4269.347309] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4269.654313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4274.995402] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92684aaed800.ost_server_uuid 50 [ 4276.816700] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92684aaed800.ost_server_uuid in FULL state after 0 sec [ 4287.793974] Lustre: server umount lustre-OST0001 complete [ 4290.023032] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4290.031424] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.050846] LustreError: Skipped 1 previous similar message [ 4290.066103] LustreError: 140043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.116122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4294.121770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.129042] LustreError: Skipped 1 previous similar message [ 4294.144072] Lustre: Skipped 1 previous similar message [ 4294.155031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4294.162968] Lustre: Skipped 3 previous similar messages [ 4298.764697] Lustre: server umount lustre-OST0000 complete [ 4305.387073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.389501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4305.410657] Lustre: Skipped 2 previous similar messages [ 4305.436137] Lustre: Skipped 3 previous similar messages [ 4309.924556] Lustre: server umount lustre-MDT0000 complete [ 4310.503064] LustreError: 138224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4310.538830] LustreError: 138224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4314.092662] LustreError: 138204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834558 with bad export cookie 4999779637140940443 [ 4314.101569] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4314.112783] LustreError: 138204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4314.590816] Lustre: server umount lustre-MDT0001 complete [ 4337.717823] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 4341.705978] Key type lgssc unregistered [ 4341.982600] LNet: 144599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4341.991044] LNetError: 144599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4342.008771] LNet: Removed LNI 192.168.202.139@tcp [ 4342.978373] Key type .llcrypt unregistered [ 4342.984555] Key type ._llcrypt unregistered [ 4383.936725] Key type ._llcrypt registered [ 4383.943938] Key type .llcrypt registered [ 4384.049609] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 4399.630081] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 4400.900421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4400.983121] alg: No test for adler32 (adler32-zlib) [ 4402.118310] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 4402.322865] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 4403.975400] Key type lgssc registered [ 4405.257851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4444.416361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4444.466277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4445.936475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4445.987657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4446.103832] Lustre: lustre-MDT0000: new disk, initializing [ 4446.229522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4446.276668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4452.339126] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4463.547382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4463.664156] Lustre: 148754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4463.715550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4463.719343] Lustre: Skipped 1 previous similar message [ 4463.794787] Lustre: lustre-MDT0001: new disk, initializing [ 4463.888621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4463.935570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4463.951581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4469.293771] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4478.739633] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4484.167801] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4492.157507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4492.587727] Lustre: lustre-OST0000: new disk, initializing [ 4492.593336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4492.600403] Lustre: 150723:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4492.730465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4501.041953] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4502.122908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4502.140744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4502.352001] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4513.074955] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4517.030337] Lustre: Failing over lustre-OST0000 [ 4517.134289] Lustre: server umount lustre-OST0000 complete [ 4534.239163] Lustre: 146978:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834762/real 1788834762] req@00000000e6d3de43 x1875729077799424/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788834778 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4534.244897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.274814] Lustre: 146978:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4534.305235] Lustre: Skipped 1 previous similar message [ 4535.270186] Lustre: 146977:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834763/real 1788834763] req@00000000628d9d7d x1875729077799680/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788834779 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4535.303857] Lustre: 146977:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4536.756071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4537.021566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.033812] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4539.046520] Lustre: 146977:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834767/real 1788834767] req@0000000095e32613 x1875729077800704/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788834783 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4544.170350] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4544.656923] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4545.183145] Lustre: 146977:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834772/real 1788834772] req@00000000faab0d34 x1875729077801856/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788834788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4545.257176] Lustre: 146977:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4545.570510] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4545.574296] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4548.877047] Lustre: Failing over lustre-MDT0000 [ 4549.426674] Lustre: server umount lustre-MDT0000 complete [ 4549.603731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.617524] Lustre: Skipped 1 previous similar message [ 4550.624638] Lustre: 146978:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834777/real 1788834777] req@00000000b9914121 x1875729077803008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788834793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4554.722201] LustreError: 148765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.743281] LustreError: 148765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4555.776529] LustreError: 148761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.851543] LustreError: 148766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.892177] LustreError: 148766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4564.895126] Lustre: 146976:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788834793/real 1788834793] req@000000006c8ef508 x1875729077813632/t0(0) o400->MGC192.168.202.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788834809 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4564.921965] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4564.937544] LustreError: 149630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4564.960179] LustreError: 149630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4569.575917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4570.079914] LustreError: 148766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.101415] LustreError: 148766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4576.236953] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa0610244323f64a5 [ 4576.251436] LustreError: 152959:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.139@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000003f9fbac5) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4576.293962] Lustre: MGC192.168.202.139@tcp: Connection restored to 0@lo (at 0@lo) [ 4576.297441] Lustre: Skipped 1 previous similar message [ 4576.670184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.769465] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4581.366183] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4581.878661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4581.915201] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4581.975111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4582.077460] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4591.593733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4591.606957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.635048] Lustre: Skipped 1 previous similar message [ 4591.648630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4596.609234] Lustre: server umount lustre-OST0000 complete [ 4602.337201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.345873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4602.354793] Lustre: Skipped 1 previous similar message [ 4602.372810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4602.375931] Lustre: Skipped 2 previous similar messages [ 4607.457940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4607.474237] LustreError: 148761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4607.494054] LustreError: 148761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4607.599684] Lustre: server umount lustre-MDT0000 complete [ 4612.182381] LustreError: 148748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788834856 with bad export cookie 11556520610733450405 [ 4612.187837] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4612.200154] LustreError: 148748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4612.696160] Lustre: server umount lustre-MDT0001 complete [ 4625.709739] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 4629.333886] Key type lgssc unregistered [ 4629.798706] LNet: 154383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4629.820251] LNetError: 154383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4630.922883] LNet: Removed LNI 192.168.202.139@tcp [ 4632.177789] Key type .llcrypt unregistered [ 4632.180711] Key type ._llcrypt unregistered [ 4672.500044] Key type ._llcrypt registered [ 4672.503195] Key type .llcrypt registered [ 4672.604883] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 4673.554644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4673.587158] alg: No test for adler32 (adler32-zlib) [ 4674.559924] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 4674.786191] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 4676.463482] Key type lgssc registered [ 4678.016675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4685.718706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4685.735392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.170233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4691.647803] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4697.569566] LustreError: 155902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.594948] LustreError: 155902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4700.399218] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4702.688327] LustreError: 155901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4704.645035] LustreError: 155930:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4704.658246] LustreError: 155930:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4704.665125] LustreError: 155930:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4705.185475] Lustre: server umount lustre-MDT0000 complete [ 4720.245626] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 4729.612053] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 4764.015559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.215834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4764.258983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4764.363753] Lustre: lustre-MDT0000: new disk, initializing [ 4764.454590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4764.472560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4768.670709] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4779.520459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4779.625444] Lustre: 159236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4779.667671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4779.671179] Lustre: Skipped 1 previous similar message [ 4779.735794] Lustre: lustre-MDT0001: new disk, initializing [ 4779.847526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4779.921352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4779.945505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4784.084838] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4794.489637] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4800.699905] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4807.840466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4808.073058] Lustre: lustre-OST0000: new disk, initializing [ 4808.077102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4808.084303] Lustre: 161207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4808.180721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4814.311356] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 4815.956892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4815.978073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4816.118391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4823.758225] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4828.678300] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4830.230647] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:37:52 (1788835072) [ 5067.233459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5067.236214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5067.248656] Lustre: Skipped 1 previous similar message [ 5067.258754] Lustre: Skipped 1 previous similar message [ 5072.354659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5072.366440] Lustre: Skipped 3 previous similar messages [ 5073.362656] Lustre: server umount lustre-MDT0000 complete [ 5077.069123] LustreError: 159226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835321 with bad export cookie 9787664450367120194 [ 5077.072686] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5077.080859] LustreError: 159226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5077.472790] LustreError: 159242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5077.476380] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5077.488821] Lustre: Skipped 1 previous similar message [ 5077.492304] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5082.592191] LustreError: 161665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.594536] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5083.585433] Lustre: server umount lustre-MDT0001 complete [ 5093.855525] Lustre: server umount lustre-OST0000 complete [ 5106.294524] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 5117.531405] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5159.218800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5159.467898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5159.505067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5159.714077] Lustre: lustre-MDT0000: new disk, initializing [ 5159.838446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5159.881440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5164.810143] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5175.358014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5175.497673] Lustre: 165871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5175.544915] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5175.548117] Lustre: Skipped 1 previous similar message [ 5175.663250] Lustre: lustre-MDT0001: new disk, initializing [ 5175.765811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5175.803216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5175.825604] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5180.535534] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5190.540141] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5196.445505] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5204.600897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5205.175646] Lustre: lustre-OST0000: new disk, initializing [ 5205.181487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5205.188458] Lustre: 167837:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5205.314865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5211.606887] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5213.777873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5213.800147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5213.923959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5221.786436] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5226.343609] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5226.581331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5231.019774] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5231.276609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5234.656247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5234.668062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5234.671926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5234.685332] Lustre: Skipped 1 previous similar message [ 5234.699751] Lustre: Skipped 1 previous similar message [ 5239.814490] Lustre: server umount lustre-OST0000 complete [ 5244.903325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5244.918992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5244.934180] Lustre: Skipped 1 previous similar message [ 5258.207180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5258.634776] Lustre: server umount lustre-MDT0000 complete [ 5259.234701] LustreError: 166839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5259.275034] LustreError: 166839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5263.151565] LustreError: 165864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835507 with bad export cookie 9787664450367279024 [ 5263.167859] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5263.172486] LustreError: 165864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5263.836637] Lustre: server umount lustre-MDT0001 complete [ 5275.905402] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:45:17 (1788835517) [ 5299.071964] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 5313.051689] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5351.717952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5352.031787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5352.069980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5352.193615] Lustre: lustre-MDT0000: new disk, initializing [ 5352.281670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5352.305242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5357.413796] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5369.086091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.254403] Lustre: 172601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5369.344032] LustreError: 172612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5369.699287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5374.593055] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5386.265390] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5394.012898] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5403.919888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5404.603937] Lustre: lustre-OST0000: new disk, initializing [ 5404.614635] Lustre: Skipped 1 previous similar message [ 5404.625653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5404.637300] Lustre: Skipped 2 previous similar messages [ 5404.655636] Lustre: 174578:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5409.929879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5409.943749] Lustre: Skipped 1 previous similar message [ 5409.969506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5410.126424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5415.293920] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5428.055036] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5440.995580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5441.000190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5441.033822] LustreError: Skipped 1 previous similar message [ 5441.036760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5441.049358] Lustre: Skipped 5 previous similar messages [ 5442.988911] Lustre: server umount lustre-OST0000 complete [ 5451.233220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.244239] Lustre: Skipped 2 previous similar messages [ 5455.386376] Lustre: server umount lustre-MDT0000 complete [ 5456.352120] LustreError: 172613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5456.372734] LustreError: 172613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5459.055607] LustreError: 172593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835703 with bad export cookie 9787664450367279906 [ 5459.062424] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5459.068666] LustreError: 172593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5459.349104] Lustre: server umount lustre-MDT0001 complete [ 5469.518288] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:48:30 (1788835710) [ 5487.542483] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 5498.661313] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5540.211297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.595630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5540.646913] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5540.774690] Lustre: lustre-MDT0000: new disk, initializing [ 5540.913210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.916260] Lustre: Skipped 2 previous similar messages [ 5540.943698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5546.723284] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5558.147493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.325809] Lustre: 178906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5558.674057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5563.732353] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5574.971462] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5582.980092] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5592.142582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5592.923716] Lustre: 180884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5600.345739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5600.533691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5601.534806] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5611.691426] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5621.219703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5621.220485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5621.231879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5621.231888] Lustre: Skipped 2 previous similar messages [ 5621.236914] LustreError: Skipped 1 previous similar message [ 5621.305899] Lustre: Skipped 1 previous similar message [ 5626.343970] Lustre: server umount lustre-OST0000 complete [ 5637.765899] Lustre: server umount lustre-MDT0000 complete [ 5641.713433] LustreError: 180092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.734280] LustreError: 180092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5642.015886] LustreError: 178897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788835886 with bad export cookie 9787664450367280991 [ 5642.034835] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5642.040765] LustreError: 178897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5642.415425] Lustre: server umount lustre-MDT0001 complete [ 5654.525150] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 5657.914922] Key type lgssc unregistered [ 5658.376564] LNet: 182739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5658.381066] LNetError: 182739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5658.396293] LNet: Removed LNI 192.168.202.139@tcp [ 5659.626338] Key type .llcrypt unregistered [ 5659.629661] Key type ._llcrypt unregistered [ 5687.774632] Key type ._llcrypt registered [ 5687.784211] Key type .llcrypt registered [ 5687.914330] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5689.200092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5689.326249] alg: No test for adler32 (adler32-zlib) [ 5690.597239] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 5690.950300] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 5692.639149] Key type lgssc registered [ 5693.905315] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5702.602523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5702.626739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.983815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5709.110972] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5714.403062] LustreError: 184152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5714.435678] LustreError: 184152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5718.312284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5718.880877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5724.285919] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5733.677742] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5740.587791] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5747.982614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5748.376824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5749.460113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5754.511792] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5761.486472] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5784.545159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5784.551521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.557283] Lustre: Skipped 1 previous similar message [ 5784.559219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5787.605543] Lustre: server umount lustre-OST0000 complete [ 5794.788217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.792469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5794.809969] Lustre: Skipped 1 previous similar message [ 5794.836435] Lustre: Skipped 2 previous similar messages [ 5799.231807] Lustre: server umount lustre-MDT0000 complete [ 5799.907115] LustreError: 185125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5799.942907] LustreError: 185125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5803.914404] LustreError: 184885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836048 with bad export cookie 3540286163415792139 [ 5803.922344] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5803.925485] LustreError: 184885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5804.418841] Lustre: server umount lustre-MDT0001 complete [ 5816.309150] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 5819.537120] Key type lgssc unregistered [ 5819.814712] LNet: 187709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5819.829121] LNetError: 187709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5819.853857] LNet: Removed LNI 192.168.202.139@tcp [ 5820.890725] Key type .llcrypt unregistered [ 5820.893207] Key type ._llcrypt unregistered [ 5847.933530] Key type ._llcrypt registered [ 5847.940223] Key type .llcrypt registered [ 5848.031388] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 5849.069158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5849.186777] alg: No test for adler32 (adler32-zlib) [ 5850.343925] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 5850.593494] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 5852.359262] Key type lgssc registered [ 5853.340823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5860.616639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5860.632820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5862.170951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5867.093775] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5872.610497] LustreError: 189125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.636206] LustreError: 189125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5877.732216] LustreError: 189126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5878.012065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.489310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5883.805921] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5891.289777] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5896.401252] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5902.843206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5903.126823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5907.273225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5910.360127] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5917.280134] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5924.839676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5924.853251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.881381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5927.906445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5927.913944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5927.914635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5927.935222] Lustre: Skipped 2 previous similar messages [ 5930.465668] LustreError: 190947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5930.679752] Lustre: server umount lustre-OST0000 complete [ 5937.727888] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9268430d1800.ost_server_uuid 50 [ 5939.280345] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5954.408638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5954.417768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5954.420698] Lustre: Skipped 1 previous similar message [ 5958.432840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5958.444913] Lustre: Skipped 1 previous similar message [ 5960.811508] Lustre: server umount lustre-MDT0000 complete [ 5963.746427] LustreError: 189125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5963.767815] LustreError: 189125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5965.170479] LustreError: 189108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836209 with bad export cookie 1334642114323138505 [ 5965.174336] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5965.178300] LustreError: 189108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5965.712233] Lustre: server umount lustre-MDT0001 complete [ 5974.828724] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:56:57 (1788836217) [ 5983.063135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.698253] LustreError: 193042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5983.769536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5989.610519] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 5992.939374] LustreError: 193043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.955316] LustreError: 193043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6000.814631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.226236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6007.488302] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6018.322486] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6026.116192] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6032.303978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6032.595826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6039.079526] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6045.913794] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6052.636174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6052.868947] Lustre: lustre-OST0001: new disk, initializing [ 6052.876504] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6052.886215] Lustre: 195884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6053.007988] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6054.106760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6059.833657] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6060.076595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6060.087860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6060.131902] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6068.860406] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6074.859787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6074.886300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.889112] Lustre: lustre-OST0000: Not available for connect from 192.168.202.39@tcp (stopping) [ 6074.907249] Lustre: Skipped 1 previous similar message [ 6078.031064] Lustre: server umount lustre-OST0000 complete [ 6079.980735] LustreError: 194866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6080.013207] LustreError: 194866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6085.262743] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6085.515564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6090.577321] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6090.821879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6106.085739] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6106.092151] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6106.106814] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6106.131322] Lustre: Skipped 2 previous similar messages [ 6106.153491] Lustre: Skipped 4 previous similar messages [ 6111.307407] Lustre: server umount lustre-OST0001 complete [ 6116.320486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.453620] Lustre: server umount lustre-MDT0000 complete [ 6126.044718] LustreError: 193023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836370 with bad export cookie 1334642114323139576 [ 6126.058923] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6126.059859] LustreError: 193023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6126.367389] Lustre: server umount lustre-MDT0001 complete [ 6154.569050] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:59:56 (1788836396) [ 6164.758314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6165.274390] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6165.380989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6165.665791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6170.702886] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6181.098929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6181.301748] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6181.385044] Lustre: 199322:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6187.745863] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6198.007494] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6206.975740] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6216.512983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6216.814820] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6225.622552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6225.707739] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6234.449937] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6241.748888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.898500] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6242.207679] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6242.210114] Lustre: Skipped 2 previous similar messages [ 6249.317965] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6251.150909] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6257.762927] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6261.734470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6261.743597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6261.762875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6261.769062] Lustre: Skipped 2 previous similar messages [ 6261.776087] Lustre: Skipped 4 previous similar messages [ 6266.866311] LustreError: 201174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6266.901569] LustreError: 201174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [ 6267.094762] Lustre: server umount lustre-OST0000 complete [ 6286.820768] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6286.836289] LustreError: Skipped 1 previous similar message [ 6287.378322] Lustre: server umount lustre-OST0001 complete [ 6297.568678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.574226] Lustre: Skipped 5 previous similar messages [ 6302.874228] Lustre: server umount lustre-MDT0000 complete [ 6307.725551] LustreError: 200079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836551 with bad export cookie 1334642114323140976 [ 6307.729733] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6307.737413] LustreError: 200079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6308.394616] Lustre: server umount lustre-MDT0001 complete [ 6337.165805] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:02:59 (1788836579) [ 6356.479217] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 6367.900831] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 6408.266428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6408.518242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6408.554569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6408.630625] Lustre: lustre-MDT0000: new disk, initializing [ 6408.697519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6408.721944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6414.120191] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6425.234696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6425.360281] Lustre: 207433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6425.433232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6425.439322] Lustre: Skipped 1 previous similar message [ 6425.529518] Lustre: lustre-MDT0001: new disk, initializing [ 6425.721109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6425.735302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6430.015098] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6440.577984] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6447.150680] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6456.611252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6457.057669] Lustre: lustre-OST0000: new disk, initializing [ 6457.064330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6457.073152] Lustre: 209404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6465.358420] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6465.627397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6465.635194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6465.815851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6474.179497] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6479.468800] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6479.742768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6486.149683] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6486.587441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6491.619108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6491.634348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6491.637688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6491.651180] Lustre: Skipped 4 previous similar messages [ 6491.670204] Lustre: Skipped 3 previous similar messages [ 6502.880422] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6503.094461] Lustre: server umount lustre-OST0000 complete [ 6513.709360] Lustre: server umount lustre-MDT0000 complete [ 6517.217153] LustreError: 207441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6517.224192] LustreError: 207441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6518.092798] LustreError: 207426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836762 with bad export cookie 1334642114323142761 [ 6518.102590] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6518.789855] Lustre: server umount lustre-MDT0001 complete [ 6529.899184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6536.302212] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6545.364041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.098458] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6559.119415] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6564.766622] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6571.899220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6579.019814] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6586.255742] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6591.664682] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6591.907234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6596.798820] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6597.035794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6600.173567] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6600.187654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6600.187665] Lustre: Skipped 2 previous similar messages [ 6600.204523] LustreError: Skipped 2 previous similar messages [ 6605.597734] Lustre: server umount lustre-OST0000 complete [ 6614.301575] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6620.083723] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6650.221812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6655.788824] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6655.813191] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6655.824624] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6655.839292] Lustre: Skipped 1 previous similar message [ 6657.334521] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6664.978953] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6668.740852] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6676.452353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.489583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6676.508124] Lustre: Skipped 12 previous similar messages [ 6680.462129] Lustre: server umount lustre-OST0000 complete [ 6690.999133] Lustre: server umount lustre-MDT0000 complete [ 6691.819388] LustreError: 211622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6691.840823] LustreError: 211622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6695.117226] LustreError: 212359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788836939 with bad export cookie 1334642114323143650 [ 6695.122271] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6695.132618] LustreError: 212359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6695.618282] Lustre: server umount lustre-MDT0001 complete [ 6706.423658] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:09:08 (1788836948) [ 6714.041137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6714.811414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6714.818711] Lustre: Skipped 6 previous similar messages [ 6719.523331] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6728.476578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6733.322284] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6741.913403] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6749.324254] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6757.135817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6765.111335] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6772.936697] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6778.383956] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6778.641845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6783.633503] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6783.912183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6791.939773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6792.252034] Lustre: lustre-OST0001: new disk, initializing [ 6792.270981] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6792.283571] Lustre: 220963:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6799.075783] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6802.590256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6802.601733] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6802.799110] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6809.699903] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6814.571478] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6814.911494] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6820.210198] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6820.520805] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6823.417302] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6823.418427] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6823.425965] Lustre: Skipped 6 previous similar messages [ 6823.444923] LustreError: Skipped 2 previous similar messages [ 6828.858841] Lustre: server umount lustre-OST0001 complete [ 6835.151792] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6835.356228] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6839.944711] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6840.208946] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6868.127626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6873.587668] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6873.605827] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6873.606809] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6874.259525] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6881.737873] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6883.219943] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6891.415417] Lustre: server umount lustre-OST0001 complete [ 6924.256254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6924.619739] Lustre: server umount lustre-OST0000 complete [ 6934.759364] Lustre: server umount lustre-MDT0000 complete [ 6938.233618] LustreError: 217572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837182 with bad export cookie 1334642114323144616 [ 6938.239312] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6938.247250] LustreError: 217572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6944.731912] Lustre: server umount lustre-MDT0001 complete [ 6968.316028] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:13:30 (1788837210) [ 6976.103537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.523644] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6976.554562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6981.409900] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 6989.838373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6989.954504] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6989.998158] Lustre: 226207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6994.282991] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7002.324118] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7008.205713] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7015.706462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7015.890345] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7022.539534] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7030.369952] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7038.965940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7039.076581] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7045.915144] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7053.436665] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7055.029727] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7060.021498] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7060.253875] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7064.767744] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7065.002668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7070.208979] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff926849059000.ost_server_uuid 50 [ 7071.748635] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff926849059000.ost_server_uuid in IDLE state after 0 sec [ 7072.690797] Lustre: Permanently deactivating lustre-OST0001 [ 7072.712150] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7075.327046] Lustre: Permanently reactivating lustre-OST0001 [ 7082.467450] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7082.475997] LustreError: Skipped 5 previous similar messages [ 7082.485780] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.498676] Lustre: Skipped 6 previous similar messages [ 7082.502259] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7082.509073] Lustre: Skipped 18 previous similar messages [ 7085.377027] Lustre: server umount lustre-OST0001 complete [ 7088.104331] LustreError: 228054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7088.118222] LustreError: 228054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 7095.067218] Lustre: server umount lustre-OST0000 complete [ 7104.819385] Lustre: server umount lustre-MDT0000 complete [ 7107.765121] LustreError: 228068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837351 with bad export cookie 1334642114323145904 [ 7107.776455] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7107.783871] LustreError: 228068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7108.012682] Lustre: server umount lustre-MDT0001 complete [ 7129.937350] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:16:12 (1788837372) [ 7137.229963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.584853] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7137.662482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7137.672906] Lustre: Skipped 5 previous similar messages [ 7142.088274] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7150.596675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.724410] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7150.799855] Lustre: 232898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7155.191255] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7162.082797] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7167.954585] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7176.022164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7176.213161] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7176.239851] Lustre: Permanently deactivating lustre-OST0000 [ 7176.252372] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7184.429711] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7184.476613] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7192.307666] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7199.612937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7199.677515] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7206.796343] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7209.727088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7214.055537] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7215.912195] Lustre: Permanently reactivating lustre-OST0000 [ 7215.930619] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7215.937201] Lustre: Skipped 2 previous similar messages [ 7217.512104] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7217.512737] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 33 seconds. I think it's dead, and I am evicting it. exp 0000000013e6655c, cur 1788837461 deadline 1788837458 last 1788837428 [ 7217.528946] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7217.558560] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7217.565023] Lustre: Skipped 1 previous similar message [ 7217.645499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7229.357318] Lustre: server umount lustre-OST0001 complete [ 7247.327151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7247.483744] Lustre: server umount lustre-OST0000 complete [ 7257.441347] Lustre: server umount lustre-MDT0000 complete [ 7261.312322] LustreError: 232889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837505 with bad export cookie 1334642114323147556 [ 7261.315335] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7261.325512] LustreError: 232889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7261.677818] Lustre: server umount lustre-MDT0001 complete [ 7274.467048] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 7276.931396] Key type lgssc unregistered [ 7277.177799] LNet: 237760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7277.183437] LNetError: 237760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7277.191854] LNet: Removed LNI 192.168.202.139@tcp [ 7277.819606] Key type .llcrypt unregistered [ 7277.821850] Key type ._llcrypt unregistered [ 7303.750984] Key type ._llcrypt registered [ 7303.753492] Key type .llcrypt registered [ 7303.900726] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 7305.263303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7305.336213] alg: No test for adler32 (adler32-zlib) [ 7306.562989] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 7306.967120] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 7308.735168] Key type lgssc registered [ 7309.926063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7320.174869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7320.209404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.850079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7326.106025] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7331.305772] LustreError: 239238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7331.347100] LustreError: 239238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7336.328389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.425865] LustreError: 239239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.573373] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7336.620665] Lustre: 239227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7336.635744] Lustre: Permanently deactivating lustre-MDT0001 [ 7336.655411] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7337.076607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7338.187814] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7338.198209] LustreError: 239270:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7338.213403] LustreError: 239270:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7338.228979] LustreError: 239270:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7342.868843] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7351.762313] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7358.691523] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7366.060985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7366.608487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.593852] LustreError: 241081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7367.617699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7372.769081] LustreError: 241081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.603140] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7377.908430] LustreError: 241130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7381.536742] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7382.010613] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7387.112513] LustreError: 241130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7387.137296] LustreError: 241130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7388.316280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7388.469185] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7393.794826] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7393.804667] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7394.677216] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7401.204615] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7404.995767] Lustre: Permanently reactivating lustre-MDT0001 [ 7405.023052] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7405.029564] Lustre: Skipped 2 previous similar messages [ 7408.081092] Lustre: lustre-MDT0001: Client c8267226-b021-4d41-8d02-6399ce325ade (at 192.168.202.39@tcp) reconnecting [ 7413.357172] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7414.583947] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7419.530462] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7419.752323] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7424.008695] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7424.239441] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7426.619233] Lustre: Permanently deactivating lustre-MDT0001 [ 7426.636424] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7426.640893] Lustre: Skipped 2 previous similar messages [ 7432.841110] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7439.329952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7439.332378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7439.358401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7439.841633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7439.859094] Lustre: Skipped 1 previous similar message [ 7439.866210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7439.875616] Lustre: Skipped 2 previous similar messages [ 7440.598031] Lustre: server umount lustre-MDT0000 complete [ 7443.845870] LustreError: 239220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837688 with bad export cookie 15389680460498058185 [ 7443.848449] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7443.860052] LustreError: 239220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7444.089668] Lustre: server umount lustre-MDT0001 complete [ 7448.020693] Lustre: server umount lustre-OST0000 complete [ 7459.223916] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:21:41 (1788837701) [ 7469.885912] Lustre: server umount lustre-OST0001 complete [ 7476.072371] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 7485.968408] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 7523.107879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.331600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7523.354321] Lustre: Skipped 2 previous similar messages [ 7523.387789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7523.490952] Lustre: lustre-MDT0000: new disk, initializing [ 7523.575281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7523.629539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7528.189853] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7537.957784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.069580] Lustre: 247495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7538.111297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7538.114865] Lustre: Skipped 1 previous similar message [ 7538.173409] Lustre: lustre-MDT0001: new disk, initializing [ 7538.274731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7538.309802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7538.319770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7542.136641] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7551.117955] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7556.272202] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7563.011589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7563.417162] Lustre: lustre-OST0000: new disk, initializing [ 7563.424384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7563.439989] Lustre: 249460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7563.568110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7569.589357] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7573.075789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7573.098185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7573.269952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7579.579402] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7581.775742] LustreError: 247502:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7591.011478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7591.253342] Lustre: lustre-OST0001: new disk, initializing [ 7591.260661] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7591.273532] Lustre: 250577:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7591.827274] LustreError: 247502:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7591.876789] LustreError: 247501:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7600.165959] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7600.823423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7600.852087] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7600.982070] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7601.943083] LustreError: 247501:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7611.908056] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7614.197416] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7616.484059] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.508072] Lustre: Skipped 1 previous similar message [ 7616.508226] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7616.515571] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7621.608428] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7621.617683] Lustre: Skipped 2 previous similar messages [ 7622.338068] Lustre: server umount lustre-OST0001 complete [ 7626.738357] LustreError: 249445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7626.773432] LustreError: 249445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7640.032805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7640.039576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7640.040204] LustreError: Skipped 1 previous similar message [ 7640.057098] Lustre: Skipped 1 previous similar message [ 7640.059480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7640.077467] Lustre: Skipped 1 previous similar message [ 7644.902413] Lustre: server umount lustre-OST0000 complete [ 7650.784663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7650.789369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.798774] Lustre: Skipped 1 previous similar message [ 7650.801589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7650.803950] Lustre: Skipped 2 previous similar messages [ 7654.753289] Lustre: server umount lustre-MDT0000 complete [ 7658.782500] LustreError: 247486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788837902 with bad export cookie 15389680460498061223 [ 7658.798540] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7658.805057] LustreError: 247486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7673.313598] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7673.700257] Lustre: server umount lustre-MDT0001 complete [ 7684.086743] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 7686.572686] Key type lgssc unregistered [ 7686.891936] LNet: 252621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7686.898965] LNetError: 252621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7686.909169] LNet: Removed LNI 192.168.202.139@tcp [ 7688.041644] Key type .llcrypt unregistered [ 7688.044932] Key type ._llcrypt unregistered [ 7725.654523] Key type ._llcrypt registered [ 7725.655901] Key type .llcrypt registered [ 7725.749560] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 7726.991670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7727.012820] alg: No test for adler32 (adler32-zlib) [ 7728.129717] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 7728.371589] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 7730.047185] Key type lgssc registered [ 7730.941541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7737.780490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7737.799739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7739.050627] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7739.079543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7739.275065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7744.204444] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7753.904880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.996853] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7754.018451] Lustre: 254819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7754.316183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7759.634354] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7767.874582] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7773.804211] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7780.180245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7780.423282] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7780.592956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7786.164485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7787.667669] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7794.287694] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7853.045493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.046484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7853.054616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7853.063231] Lustre: Skipped 1 previous similar message [ 7854.513968] Lustre: server umount lustre-OST0000 complete [ 7858.876415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7870.799191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7871.176759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.251958] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7873.259322] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7873.264302] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7876.923705] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 7884.010796] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7924.704851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7924.720746] LustreError: Skipped 1 previous similar message [ 7924.722630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.743023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7924.748849] Lustre: Skipped 1 previous similar message [ 7924.767345] Lustre: Skipped 2 previous similar messages [ 7929.829875] LustreError: 258408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7929.859690] LustreError: 258408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7929.985248] Lustre: server umount lustre-OST0000 complete [ 7934.946057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7934.963170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7934.966915] Lustre: Skipped 1 previous similar message [ 7940.039193] Lustre: server umount lustre-MDT0000 complete [ 7940.068869] LustreError: 254831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7940.092252] LustreError: 254831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7944.046525] LustreError: 256689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838188 with bad export cookie 4702941571869414478 [ 7944.063870] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7944.068463] LustreError: 256689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7944.439497] Lustre: server umount lustre-MDT0001 complete [ 7954.570230] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 7957.631363] Key type lgssc unregistered [ 7958.002063] LNet: 260238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7958.008445] LNetError: 260238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7958.037353] LNet: Removed LNI 192.168.202.139@tcp [ 7958.823206] Key type .llcrypt unregistered [ 7958.825480] Key type ._llcrypt unregistered [ 7982.007567] Key type ._llcrypt registered [ 7982.010346] Key type .llcrypt registered [ 7982.095971] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 7983.100043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7983.195510] alg: No test for adler32 (adler32-zlib) [ 7984.319857] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 7984.618738] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 7986.351172] Key type lgssc registered [ 7987.784704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7994.796510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7994.818746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7996.375646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.652475] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8006.631673] LustreError: 261650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8006.649972] LustreError: 261650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8008.833599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.257501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8014.190848] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8021.473338] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8027.594605] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8034.710865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8034.811876] LustreError: 263475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8035.283394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8041.461250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8041.784489] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8049.029340] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8076.256516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8076.264256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8076.282375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.282738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8077.283604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8077.301312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8080.175277] Lustre: server umount lustre-OST0000 complete [ 8085.984922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8085.997796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8086.014849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8086.020536] Lustre: Skipped 1 previous similar message [ 8092.642680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8092.651970] Lustre: Skipped 3 previous similar messages [ 8097.762695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8097.771957] Lustre: Skipped 1 previous similar message [ 8098.272954] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8098.775351] Lustre: server umount lustre-MDT0000 complete [ 8101.871816] LustreError: 261647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8101.889992] LustreError: 261647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8102.191446] LustreError: 261630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838346 with bad export cookie 10348789931789750480 [ 8102.201406] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8102.203723] LustreError: 261630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8102.618342] Lustre: server umount lustre-MDT0001 complete [ 8114.838489] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 8118.421331] Key type lgssc unregistered [ 8118.785779] LNet: 265957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8118.799261] LNetError: 265957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8118.818492] LNet: Removed LNI 192.168.202.139@tcp [ 8119.892366] Key type .llcrypt unregistered [ 8119.894333] Key type ._llcrypt unregistered [ 8141.578835] Key type ._llcrypt registered [ 8141.584596] Key type .llcrypt registered [ 8141.716602] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8142.830074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8142.975016] alg: No test for adler32 (adler32-zlib) [ 8144.389723] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8144.719641] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8146.455381] Key type lgssc registered [ 8148.021639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8156.406831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8156.429791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8157.898882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8162.184763] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8168.421298] LustreError: 267176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8168.434415] LustreError: 267176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8170.613277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8171.083283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8175.652832] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8185.752816] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8193.096660] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8201.254867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8201.797324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8203.862279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8209.499775] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8219.406243] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8231.505788] Lustre: server umount lustre-OST0000 complete [ 8237.033761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.043484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8237.073713] Lustre: Skipped 1 previous similar message [ 8237.093332] Lustre: Skipped 1 previous similar message [ 8242.144905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8242.163804] Lustre: Skipped 1 previous similar message [ 8246.440076] Lustre: server umount lustre-MDT0000 complete [ 8247.270395] LustreError: 267177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8247.292960] LustreError: 267177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8248.288111] Lustre: 266487:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788838476/real 1788838476] req@0000000026c928e6 x1875733001565568/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788838492 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8248.321899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.357785] LustreError: 267905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8248.401506] LustreError: 267905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8251.241483] LustreError: 270104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838495 with bad export cookie 4555467771846952258 [ 8251.244059] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8251.251027] LustreError: 270104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8251.631920] Lustre: server umount lustre-MDT0001 complete [ 8264.037370] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 8267.686904] Key type lgssc unregistered [ 8268.084609] LNet: 270856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8268.114256] LNetError: 270856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8269.185465] LNet: Removed LNI 192.168.202.139@tcp [ 8270.187410] Key type .llcrypt unregistered [ 8270.190043] Key type ._llcrypt unregistered [ 8297.097393] Key type ._llcrypt registered [ 8297.104846] Key type .llcrypt registered [ 8297.254227] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8298.192350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8298.262853] alg: No test for adler32 (adler32-zlib) [ 8299.407957] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8299.770709] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8301.495185] Key type lgssc registered [ 8303.598751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8313.163509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8313.200569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8314.814828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8321.355160] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8324.082677] LustreError: 272268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8324.103614] LustreError: 272268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8329.186487] LustreError: 272269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8331.263033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8331.440986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8336.941436] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8345.026740] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8350.893303] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8357.868302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8358.276271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8359.297965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8364.457843] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8372.044439] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8402.406239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8402.414763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.425691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8402.436512] Lustre: Skipped 1 previous similar message [ 8407.163888] Lustre: server umount lustre-OST0000 complete [ 8412.642957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8412.645914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8412.668187] Lustre: Skipped 2 previous similar messages [ 8417.498119] Lustre: server umount lustre-MDT0000 complete [ 8417.760801] LustreError: 272268:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8417.786851] LustreError: 272268:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8421.237743] LustreError: 272249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838665 with bad export cookie 11326847272797553860 [ 8421.245405] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8421.245452] LustreError: 272249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8421.453914] Lustre: server umount lustre-MDT0001 complete [ 8432.902160] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 8436.036473] Key type lgssc unregistered [ 8436.305069] LNet: 276576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8436.315786] LNetError: 276576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8436.340904] LNet: Removed LNI 192.168.202.139@tcp [ 8437.500402] Key type .llcrypt unregistered [ 8437.502239] Key type ._llcrypt unregistered [ 8458.204470] Key type ._llcrypt registered [ 8458.206707] Key type .llcrypt registered [ 8458.434811] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8459.633498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8459.685108] alg: No test for adler32 (adler32-zlib) [ 8461.053767] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8461.492109] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8463.287883] Key type lgssc registered [ 8464.772154] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8475.468064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8475.491301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8477.322046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.183414] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8486.887877] LustreError: 277810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8486.920917] LustreError: 277810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8492.004957] LustreError: 277812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.301949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.739956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8500.543774] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8508.569503] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8517.424955] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8525.841815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8526.373134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8532.890861] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8540.306849] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8546.866584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8562.145691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8562.174820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8564.334191] Lustre: server umount lustre-OST0000 complete [ 8571.360512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8571.369166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.387640] Lustre: Skipped 1 previous similar message [ 8571.397080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8571.409775] Lustre: Skipped 1 previous similar message [ 8572.386563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8572.403749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8572.409967] Lustre: Skipped 1 previous similar message [ 8574.675456] Lustre: server umount lustre-MDT0000 complete [ 8577.505735] LustreError: 277792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8577.547495] LustreError: 277792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8579.233215] LustreError: 277776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788838823 with bad export cookie 16286939241697942282 [ 8579.245022] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8579.249728] LustreError: 277776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8579.853741] Lustre: server umount lustre-MDT0001 complete [ 8592.582946] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 8595.186421] Key type lgssc unregistered [ 8595.500437] LNet: 281530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8595.527781] LNetError: 281530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8595.551986] LNet: Removed LNI 192.168.202.139@tcp [ 8596.460196] Key type .llcrypt unregistered [ 8596.463947] Key type ._llcrypt unregistered [ 8608.147488] Key type ._llcrypt registered [ 8608.149541] Key type .llcrypt registered [ 8608.266477] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8625.543889] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 8639.657744] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8640.689705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8640.755344] alg: No test for adler32 (adler32-zlib) [ 8641.934913] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8642.259096] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8643.959232] Key type lgssc registered [ 8644.893544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8672.793691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8672.816380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8674.078556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8674.147242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8674.256813] Lustre: lustre-MDT0000: new disk, initializing [ 8674.334354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8674.351207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8677.802605] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8687.265771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8687.383245] Lustre: 285075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8687.463306] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8687.468268] Lustre: Skipped 1 previous similar message [ 8687.537390] Lustre: lustre-MDT0001: new disk, initializing [ 8687.614106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8687.640892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8687.654046] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8692.621211] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8702.154723] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8708.320072] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8715.055549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8715.443815] Lustre: lustre-OST0000: new disk, initializing [ 8715.447521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8715.468854] Lustre: 287043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8715.610575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8721.468201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8721.486780] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8721.593699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8721.900722] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8730.855899] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8736.315769] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8736.593669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8741.566564] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8741.769182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8747.493616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8747.497072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8747.512312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8747.531628] Lustre: Skipped 1 previous similar message [ 8749.883403] Lustre: server umount lustre-OST0000 complete [ 8754.148500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8754.156097] LustreError: Skipped 1 previous similar message [ 8754.169047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.179953] Lustre: Skipped 1 previous similar message [ 8754.188698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8757.728269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8757.729478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.072175] Lustre: server umount lustre-MDT0000 complete [ 8763.534191] LustreError: 285069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839007 with bad export cookie 13766153595765237795 [ 8763.543569] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8763.547026] LustreError: 285069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8763.773952] Lustre: server umount lustre-MDT0001 complete [ 8771.174739] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:43:33 (1788839013) [ 8776.569273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8776.968153] LustreError: 289459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.991528] LustreError: 289459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8777.053957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.165892] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8782.306595] LustreError: 289460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8782.327284] LustreError: 289460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8787.426203] LustreError: 289459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8788.362542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8788.566815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8792.362532] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8798.971603] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8803.751191] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8810.050388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8810.426622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8815.426957] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8824.156379] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8832.996577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8833.002601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8833.007811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8833.007821] Lustre: Skipped 1 previous similar message [ 8833.029417] Lustre: Skipped 1 previous similar message [ 8838.114736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8838.126105] Lustre: Skipped 1 previous similar message [ 8839.064525] Lustre: server umount lustre-OST0000 complete [ 8843.235086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8848.353420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8848.357334] Lustre: Skipped 3 previous similar messages [ 8848.693889] Lustre: server umount lustre-MDT0000 complete [ 8851.863063] LustreError: 289438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839096 with bad export cookie 13766153595765238677 [ 8851.865704] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8851.871231] LustreError: 289438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8852.171755] Lustre: server umount lustre-MDT0001 complete [ 8860.210226] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 8862.230524] Key type lgssc unregistered [ 8862.504619] LNet: 292992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8862.512882] LNetError: 292992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8862.526086] LNet: Removed LNI 192.168.202.139@tcp [ 8863.323291] Key type .llcrypt unregistered [ 8863.326534] Key type ._llcrypt unregistered [ 8887.051630] Key type ._llcrypt registered [ 8887.053525] Key type .llcrypt registered [ 8887.150965] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 8899.726945] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 8900.985950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8901.051448] alg: No test for adler32 (adler32-zlib) [ 8902.022981] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 8902.179586] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 8903.823288] Key type lgssc registered [ 8904.984927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8949.742743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8949.785578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8951.026289] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8951.052252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8951.197092] Lustre: lustre-MDT0000: new disk, initializing [ 8951.317038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8951.375860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8957.151084] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8970.649663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8970.902088] Lustre: 296793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8971.027117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8971.036543] Lustre: Skipped 1 previous similar message [ 8971.253461] Lustre: lustre-MDT0001: new disk, initializing [ 8971.513993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8971.614525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8971.647085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8977.591675] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 8988.369375] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8995.229435] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9003.268252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9003.647341] Lustre: lustre-OST2710: new disk, initializing [ 9003.659745] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9003.681086] Lustre: 298760:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9003.841384] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9011.565938] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9013.314054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9013.326738] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9013.395712] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9021.229876] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9029.030574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9029.273653] Lustre: lustre-OST03e8: new disk, initializing [ 9029.278110] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9029.283687] Lustre: 299825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9029.362218] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9034.912496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9034.931857] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9035.055911] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9037.316588] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9046.795660] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9055.437728] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9055.701580] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9061.847858] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9062.151313] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9066.987811] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9067.243500] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9073.312368] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9073.689212] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9079.279787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9079.292422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9079.343242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9080.525411] Lustre: server umount lustre-MDT0000 complete [ 9081.331093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9085.197571] LustreError: 296787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839329 with bad export cookie 4731734198378500244 [ 9085.202860] LustreError: 296787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9085.206730] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9085.836032] Lustre: server umount lustre-MDT0001 complete [ 9100.425557] Lustre: server umount lustre-OST2710 complete [ 9102.303150] Lustre: 294592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839330/real 1788839330] req@00000000b3e52833 x1875733796427136/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788839346 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9102.324909] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.336129] Lustre: Skipped 2 previous similar messages [ 9106.353192] Lustre: 294590:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788839334/real 1788839334] req@000000008ee841f4 x1875733796427392/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788839350 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9107.119218] Lustre: server umount lustre-OST03e8 complete [ 9127.460380] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [ 9137.895874] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9175.093524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9175.391657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9175.423989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9175.512782] Lustre: lustre-MDT0000: new disk, initializing [ 9175.626325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9175.653049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9180.512685] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9191.846253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.963183] Lustre: 305075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9192.021441] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9192.025307] Lustre: Skipped 1 previous similar message [ 9192.102992] Lustre: lustre-MDT0001: new disk, initializing [ 9192.203503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9192.230045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9192.245801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9197.796240] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9207.166200] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9213.806244] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9221.490504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9221.771586] Lustre: lustre-OST0000: new disk, initializing [ 9221.775296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9221.784764] Lustre: 307044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9221.899461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9228.005184] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9229.886154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9229.899210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9230.018832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9237.222947] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9241.255931] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9241.407243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9246.159754] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9246.431866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9250.784673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9250.791755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9250.794870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9250.817291] Lustre: Skipped 1 previous similar message [ 9254.694803] Lustre: server umount lustre-OST0000 complete [ 9258.976983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9258.986066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9259.006342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9259.012035] Lustre: Skipped 1 previous similar message [ 9261.029858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9264.531087] Lustre: server umount lustre-MDT0000 complete [ 9266.150565] LustreError: 307344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9266.164951] LustreError: 307344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9267.939099] LustreError: 305870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839512 with bad export cookie 4731734198378501924 [ 9267.946801] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9267.951685] LustreError: 305870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9268.237536] Lustre: server umount lustre-MDT0001 complete [ 9276.625550] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:51:59 (1788839519) [ 9278.234436] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9279.870238] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:52:02 (1788839522) [ 9284.194715] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9312.810293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9313.078225] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9313.101338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9313.287183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9318.007196] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9328.233748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.356963] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9328.397391] Lustre: 310902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9334.813555] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9357.942842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9358.253686] Lustre: Denying initial registration attempt from nid 192.168.202.139@tcp, specified as failover [ 9358.274501] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9358.283709] LustreError: 312355:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9358.290830] LustreError: 312355:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9358.311340] LustreError: 312355:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9358.425625] Lustre: server umount lustre-OST0000 complete [ 9358.432576] LustreError: 312355:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9369.569804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9369.570554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9369.601204] Lustre: Skipped 1 previous similar message [ 9369.617855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9369.627718] Lustre: Skipped 2 previous similar messages [ 9371.075345] Lustre: server umount lustre-MDT0000 complete [ 9374.689074] LustreError: 310907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9374.702086] LustreError: 310907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9374.900594] LustreError: 310894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839619 with bad export cookie 4731734198378502806 [ 9374.902366] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9374.912514] LustreError: 310894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9375.332073] Lustre: server umount lustre-MDT0001 complete [ 9385.478470] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 9388.227569] Key type lgssc unregistered [ 9388.611926] LNet: 313476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9388.643961] LNetError: 313476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9388.684362] LNet: Removed LNI 192.168.202.139@tcp [ 9389.499747] Key type .llcrypt unregistered [ 9389.502556] Key type ._llcrypt unregistered [ 9401.106432] Key type ._llcrypt registered [ 9401.109457] Key type .llcrypt registered [ 9401.205329] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9402.533137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9402.567308] alg: No test for adler32 (adler32-zlib) [ 9403.610622] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 9403.880523] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 9405.615414] Key type lgssc registered [ 9406.911467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9452.453448] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9460.377658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9460.392558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.632688] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9461.651230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9461.824202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.208895] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9474.346108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9474.510872] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9474.547340] Lustre: 316241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9474.804372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9479.079925] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9486.388487] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9491.761907] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9511.092126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9511.264850] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9511.420889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9518.141169] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9525.542032] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9531.874387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9531.876846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9531.879328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9531.880207] Lustre: Skipped 1 previous similar message [ 9531.891779] LustreError: Skipped 1 previous similar message [ 9534.452629] Lustre: server umount lustre-OST0000 complete [ 9541.602916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9541.610755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9541.623983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9541.638464] Lustre: Skipped 1 previous similar message [ 9544.552591] Lustre: server umount lustre-MDT0000 complete [ 9548.065544] LustreError: 316232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839792 with bad export cookie 3291589995440327456 [ 9548.074561] LustreError: 316232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9548.076751] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9548.370971] Lustre: server umount lustre-MDT0001 complete [ 9558.705488] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 9561.486444] Key type lgssc unregistered [ 9561.721817] LNet: 319814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9561.726328] LNetError: 319814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9561.737953] LNet: Removed LNI 192.168.202.139@tcp [ 9562.501457] Key type .llcrypt unregistered [ 9562.505929] Key type ._llcrypt unregistered [ 9584.447788] Key type ._llcrypt registered [ 9584.453393] Key type .llcrypt registered [ 9584.523916] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9585.622467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9585.679922] alg: No test for adler32 (adler32-zlib) [ 9586.731359] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 9587.030472] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 9588.703286] Key type lgssc registered [ 9589.928664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9597.398257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9597.439034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9599.001140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9603.412794] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9609.189710] LustreError: 321207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9609.207513] LustreError: 321207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9610.928145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9611.201351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9614.726164] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9621.054523] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9627.020815] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9634.797776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9635.040972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9640.928790] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9648.059555] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9657.312469] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9657.325639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9657.342393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9659.878176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9659.888756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9659.891792] Lustre: Skipped 2 previous similar messages [ 9661.333176] Lustre: server umount lustre-MDT0000 complete [ 9664.999873] LustreError: 321202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9665.028688] LustreError: 321202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9665.171211] LustreError: 321189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788839909 with bad export cookie 9920425473349300767 [ 9665.195149] LustreError: 321189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9665.199908] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9670.115212] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9670.119244] LustreError: 322616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9670.129436] Lustre: Skipped 1 previous similar message [ 9670.132745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9670.145245] LustreError: 322616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9670.173733] Lustre: Skipped 2 previous similar messages [ 9671.549297] Lustre: server umount lustre-MDT0001 complete [ 9677.306654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9685.937121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9691.559979] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x89ac6e6847b8cfa0 [ 9691.584884] Lustre: MGC192.168.202.139@tcp: Connection restored to 0@lo (at 0@lo) [ 9692.103918] LustreError: 323039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9692.252644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9692.380593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9697.124354] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9700.322530] LustreError: 323040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9700.344214] LustreError: 323040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9701.350559] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9707.315924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.835760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9709.894707] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9713.534718] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9724.859593] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9727.291798] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9734.462142] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9736.667640] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9743.328626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9743.333758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.343404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9751.011667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9751.018887] Lustre: Skipped 4 previous similar messages [ 9756.128490] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9756.345446] Lustre: server umount lustre-OST0000 complete [ 9761.261628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9761.288195] Lustre: Skipped 1 previous similar message [ 9761.294681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9761.306430] Lustre: Skipped 2 previous similar messages [ 9766.369337] LustreError: 324805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.383454] LustreError: 324805:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9766.458027] Lustre: server umount lustre-MDT0000 complete [ 9770.094123] LustreError: 321187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840014 with bad export cookie 9920425473349308320 [ 9770.098058] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9770.106128] LustreError: 321187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9770.426196] Lustre: server umount lustre-MDT0001 complete [ 9780.935436] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [ 9783.681520] Key type lgssc unregistered [ 9783.976973] LNet: 327480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9783.996337] LNetError: 327480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9784.016545] LNet: Removed LNI 192.168.202.139@tcp [ 9784.821307] Key type .llcrypt unregistered [ 9784.823313] Key type ._llcrypt unregistered [ 9808.640885] Key type ._llcrypt registered [ 9808.649990] Key type .llcrypt registered [ 9808.723996] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [ 9810.106227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9810.129591] alg: No test for adler32 (adler32-zlib) [ 9811.173875] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [ 9811.476220] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [ 9813.151203] Key type lgssc registered [ 9814.215771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9822.260276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9822.277160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9823.720845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9828.477957] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9832.934586] LustreError: 328906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9832.945582] LustreError: 328906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9836.495578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9836.793535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9840.490915] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9846.953711] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9854.010273] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9857.511994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9857.527090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9857.543746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9858.529539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9858.534408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9858.544352] Lustre: Skipped 1 previous similar message [ 9861.606356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9861.612518] Lustre: Skipped 1 previous similar message [ 9866.726186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9866.736482] Lustre: Skipped 1 previous similar message [ 9871.327165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9871.597923] Lustre: server umount lustre-MDT0000 complete [ 9871.846611] LustreError: 329634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9874.997334] LustreError: 328886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840119 with bad export cookie 15142051252764499368 [ 9874.999600] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9875.005649] LustreError: 328886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9875.299675] Lustre: server umount lustre-MDT0001 complete [ 9889.878040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9890.167102] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9890.197526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9890.383215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9894.547142] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9902.694830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9902.758151] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9902.780760] Lustre: 331495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9903.050463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9907.241392] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9915.331779] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9920.889476] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9931.441720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9932.042129] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9932.079556] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9932.093084] LustreError: 333378:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 9932.114198] LustreError: 333378:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9932.126592] LustreError: 333378:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9932.294263] Lustre: server umount lustre-OST0000 complete [ 9932.303753] LustreError: 333378:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9944.072512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9944.307220] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9944.693704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9951.300287] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9952.454680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9959.825232] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9968.302079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9968.394925] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9968.507518] Lustre: lustre-OST0001: new disk, initializing [ 9968.514893] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9968.521602] Lustre: 334921:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9968.626698] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9976.242948] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [ 9976.432896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9976.445885] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9976.570270] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9985.022984] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9992.170196] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9992.172413] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9992.175670] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9992.192499] Lustre: Skipped 1 previous similar message [ 9994.469295] Lustre: server umount lustre-OST0001 complete [ 9997.288120] LustreError: 335295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9997.303479] LustreError: 335295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9998.816742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9998.817348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9998.828945] LustreError: Skipped 2 previous similar messages [ 9998.859240] Lustre: Skipped 1 previous similar message [10002.418230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10002.430076] Lustre: Skipped 3 previous similar messages [10002.437425] LustreError: 333845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.489452] LustreError: 333845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10003.611637] Lustre: server umount lustre-OST0000 complete [10010.596868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10010.600742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10022.367255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10022.782996] Lustre: server umount lustre-MDT0000 complete [10022.882828] LustreError: 331508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10022.913072] LustreError: 331508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10026.775854] LustreError: 331487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840270 with bad export cookie 15142051252764499921 [10026.777802] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10026.795735] LustreError: 331487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10027.330674] Lustre: server umount lustre-MDT0001 complete [10039.231297] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10041.936362] Key type lgssc unregistered [10042.210709] LNet: 336880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10042.218232] LNetError: 336880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10043.257135] LNet: Removed LNI 192.168.202.139@tcp [10043.957851] Key type .llcrypt unregistered [10043.960729] Key type ._llcrypt unregistered [10111.206392] Key type ._llcrypt registered [10111.207896] Key type .llcrypt registered [10111.300658] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [10127.064630] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10128.547903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10128.870853] alg: No test for adler32 (adler32-zlib) [10129.927931] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [10130.181164] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10131.887254] Key type lgssc registered [10133.472717] Lustre: Echo OBD driver; http://www.lustre.org/ [10171.168918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10171.202639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10172.500513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10172.520078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10172.583611] Lustre: lustre-MDT0000: new disk, initializing [10172.658368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10172.673752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10177.901686] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10190.073897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10190.196520] Lustre: 341813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10190.229828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10190.232060] Lustre: Skipped 1 previous similar message [10190.285200] Lustre: lustre-MDT0001: new disk, initializing [10190.378858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10190.420949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10190.433450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10195.029557] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10207.135055] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10212.814869] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10220.445543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10220.776309] Lustre: lustre-OST0000: new disk, initializing [10220.783126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10220.788844] Lustre: 343788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10220.921368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10226.985521] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10229.357588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10229.369631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10229.518056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10237.117811] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10242.225477] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10242.490897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10247.326306] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10247.558223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10250.209957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10250.210975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10250.218386] LustreError: Skipped 1 previous similar message [10250.227083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10250.244420] Lustre: Skipped 1 previous similar message [10250.269126] Lustre: Skipped 1 previous similar message [10255.339138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10255.345444] Lustre: Skipped 1 previous similar message [10255.598975] Lustre: server umount lustre-OST0000 complete [10260.450945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.453096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10260.470141] Lustre: Skipped 1 previous similar message [10264.753071] Lustre: server umount lustre-MDT0000 complete [10268.395733] LustreError: 343787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840512 with bad export cookie 7558931400473935538 [10268.422185] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10268.983434] Lustre: server umount lustre-MDT0001 complete [10277.251830] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 00:08:39 (1788840519) [10284.348769] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 00:08:46 (1788840526) [10294.388793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10294.895371] LustreError: 346542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10294.906435] LustreError: 346542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10295.008325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10300.390685] LustreError: 346543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10300.821516] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10305.520845] LustreError: 346542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.824730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.540450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10317.184711] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10327.248670] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10335.335733] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10342.448707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10342.823047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10349.094617] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10356.451268] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10373.127577] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10374.914154] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10376.403891] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10378.105450] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10379.884908] Lustre: Failing over lustre-MDT0000 [10380.393525] Lustre: server umount lustre-MDT0000 complete [10380.770064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10380.777804] LustreError: 346538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10380.786170] Lustre: Skipped 2 previous similar messages [10380.803021] LustreError: 346538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10385.889720] LustreError: 346543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.913230] LustreError: 346543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10396.127572] Lustre: 340037:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840624/real 1788840624] req@000000008cbe6f97 x1875735084078592/t0(0) o400->MGC192.168.202.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788840640 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10396.146207] LustreError: 347466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10396.161948] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10396.194958] LustreError: 347466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10399.324559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10406.372085] LustreError: 340035:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007966f8a3 x1875735084083840/t0(0) o250->MGC192.168.202.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10406.717283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10406.769828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10409.440822] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10410.924738] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10412.027116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10412.059893] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10412.103940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10421.043643] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10422.977424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10426.896973] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10429.008366] Lustre: Failing over lustre-MDT0000 [10429.207547] Lustre: server umount lustre-MDT0000 complete [10429.974018] LustreError: 346538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.001203] LustreError: 346538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10430.436201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.444176] Lustre: Skipped 2 previous similar messages [10446.818660] Lustre: 340036:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788840674/real 1788840674] req@0000000010e16760 x1875735084099456/t0(0) o400->MGC192.168.202.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788840690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10446.858559] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10447.931403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10457.062638] LustreError: 340035:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bcedd49a x1875735084105600/t0(0) o250->MGC192.168.202.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10457.489215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10457.555837] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10460.676503] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10462.469434] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10462.700104] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10462.703628] Lustre: Skipped 2 previous similar messages [10462.737807] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10462.814142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10472.473246] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10474.066069] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10483.173460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10483.174396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10483.204168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10483.204177] Lustre: Skipped 1 previous similar message [10486.765929] Lustre: server umount lustre-MDT0000 complete [10488.288220] LustreError: 346543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10488.310591] LustreError: 346543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [10490.580736] LustreError: 346522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840734 with bad export cookie 7558931400473938254 [10490.595558] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10490.607651] LustreError: 346522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10490.987799] Lustre: server umount lustre-MDT0001 complete [10505.856826] Lustre: server umount lustre-OST0000 complete [10522.019972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10522.446426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10526.840335] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10535.371874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10540.849995] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10549.682401] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10557.665437] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10564.676882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10567.118057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10571.487322] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10579.096039] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10590.689573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10590.704321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10590.725831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10590.737642] Lustre: Skipped 3 previous similar messages [10590.760249] Lustre: Skipped 3 previous similar messages [10595.727115] Lustre: server umount lustre-OST0000 complete [10600.948435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10606.332916] Lustre: server umount lustre-MDT0000 complete [10610.119911] LustreError: 352887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840854 with bad export cookie 7558931400473938996 [10610.122863] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10610.129517] LustreError: 352887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10610.542748] Lustre: server umount lustre-MDT0001 complete [10621.838609] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10625.271916] Key type lgssc unregistered [10625.707898] LNet: 356440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10625.717590] LNetError: 356440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10625.742370] LNet: Removed LNI 192.168.202.139@tcp [10626.791159] Key type .llcrypt unregistered [10626.792845] Key type ._llcrypt unregistered [10653.319122] Key type ._llcrypt registered [10653.327173] Key type .llcrypt registered [10653.522859] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10654.672142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10654.745722] alg: No test for adler32 (adler32-zlib) [10655.783375] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [10655.997215] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10657.639608] Key type lgssc registered [10658.605177] Lustre: Echo OBD driver; http://www.lustre.org/ [10666.991707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10667.010362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.466622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10673.435334] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10678.753250] LustreError: 357952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.768344] LustreError: 357952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10682.364960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.855477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10687.608857] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10696.988901] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10703.231468] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10712.193192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10712.630478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10713.660984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10719.147883] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10728.144574] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10739.176672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10739.187611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10739.225221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10741.414528] Lustre: server umount lustre-MDT0000 complete [10742.249106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.253057] LustreError: 357948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.261277] Lustre: Skipped 1 previous similar message [10742.280139] LustreError: 357948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10745.020586] LustreError: 357934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788840989 with bad export cookie 9720578051337056825 [10745.025112] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10745.037959] LustreError: 357934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10745.348471] Lustre: server umount lustre-MDT0001 complete [10759.176705] Lustre: server umount lustre-OST0000 complete [10772.057350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10772.470288] LustreError: 361663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10772.491279] LustreError: 361663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10772.550552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10776.989857] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10777.569735] LustreError: 361664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.696316] LustreError: 361663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10785.532317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10785.972124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10791.019756] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10798.245810] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10803.668304] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10811.661611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10811.819857] LustreError: 363486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10812.074208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10817.203827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10818.250827] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10826.140853] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10841.068584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10841.075570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10841.083016] Lustre: Skipped 2 previous similar messages [10841.090267] Lustre: Skipped 1 previous similar message [10845.523616] Lustre: server umount lustre-MDT0000 complete [10846.177215] LustreError: 361664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.221663] LustreError: 361664:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10849.698466] LustreError: 361644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841093 with bad export cookie 9720578051337057938 [10849.705180] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10849.708946] LustreError: 361644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10849.939799] Lustre: server umount lustre-MDT0001 complete [10864.909788] Lustre: server umount lustre-OST0000 complete [10880.417927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10880.897397] LustreError: 365578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10881.006817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10884.824951] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10892.449610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10897.520706] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10904.330232] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10909.836317] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10915.799344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10916.117057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10916.123570] Lustre: Skipped 1 previous similar message [10921.765996] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [10926.404508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10929.237459] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10936.800855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10936.810093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10936.818319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10936.818338] Lustre: Skipped 1 previous similar message [10936.862292] Lustre: Skipped 1 previous similar message [10939.448829] Lustre: server umount lustre-OST0000 complete [10943.968065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10943.971716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10943.987472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10943.991447] Lustre: Skipped 1 previous similar message [10949.426333] Lustre: server umount lustre-MDT0000 complete [10952.175942] LustreError: 365578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10952.191593] LustreError: 365578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10952.817285] LustreError: 365559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841196 with bad export cookie 9720578051337058960 [10952.831924] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10952.835956] LustreError: 365559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10953.198582] Lustre: server umount lustre-MDT0001 complete [10963.420377] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [10965.867444] Key type lgssc unregistered [10966.183711] LNet: 369108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10966.195030] LNetError: 369108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10966.224812] LNet: Removed LNI 192.168.202.139@tcp [10967.190250] Key type .llcrypt unregistered [10967.192247] Key type ._llcrypt unregistered [10991.318527] Key type ._llcrypt registered [10991.322855] Key type .llcrypt registered [10991.463936] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [10992.578903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10992.636092] alg: No test for adler32 (adler32-zlib) [10993.994856] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [10994.335277] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [10996.095188] Key type lgssc registered [10997.168036] Lustre: Echo OBD driver; http://www.lustre.org/ [11004.914345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11004.937800] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.945537] LustreError: 370545:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11004.970846] LustreError: 370545:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11004.987060] LustreError: 370545:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11005.004632] LustreError: 370545:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11005.009266] LustreError: 370545:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11012.183269] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11012.197397] LustreError: 370942:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11012.220613] LustreError: 370942:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11012.226031] LustreError: 370942:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11012.231155] LustreError: 370942:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11012.234646] LustreError: 370942:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11032.726346] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11035.566454] Key type lgssc unregistered [11035.871694] LNet: 371946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11035.877141] LNetError: 371946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11035.892497] LNet: Removed LNI 192.168.202.139@tcp [11036.774214] Key type .llcrypt unregistered [11036.777597] Key type ._llcrypt unregistered [11056.687426] Key type ._llcrypt registered [11056.689372] Key type .llcrypt registered [11056.757444] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [11071.393381] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [11072.177846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11072.282670] alg: No test for adler32 (adler32-zlib) [11073.290918] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [11073.593129] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [11075.351320] Key type lgssc registered [11076.474335] Lustre: Echo OBD driver; http://www.lustre.org/ [11110.977756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11111.010093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11112.317716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11112.367130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11112.484988] Lustre: lustre-MDT0000: new disk, initializing [11112.576843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11112.620230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11117.311677] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11127.534975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11127.656679] Lustre: 375099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11127.709754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11127.715106] Lustre: Skipped 1 previous similar message [11127.781132] Lustre: lustre-MDT0001: new disk, initializing [11127.911965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11127.956522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11127.969557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11132.136859] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11141.565792] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11148.036767] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11156.539950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11156.759225] Lustre: lustre-OST0000: new disk, initializing [11156.764754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11156.771396] Lustre: 377067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11156.852371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11163.268954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11163.289045] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11163.423412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11163.974950] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11175.277220] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11180.899590] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11181.121822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11186.345049] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11186.616076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11189.232699] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11189.249272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11189.262238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11189.268976] Lustre: Skipped 1 previous similar message [11189.306953] Lustre: Skipped 1 previous similar message [11194.341877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11194.346228] Lustre: Skipped 1 previous similar message [11195.338933] Lustre: server umount lustre-OST0000 complete [11204.578657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11204.584687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11204.590072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11204.602623] Lustre: Skipped 1 previous similar message [11204.636687] LustreError: Skipped 1 previous similar message [11206.286182] Lustre: server umount lustre-MDT0000 complete [11210.480959] LustreError: 375090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841454 with bad export cookie 4211604404158471211 [11210.488632] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11210.503066] LustreError: 375090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11210.834814] Lustre: server umount lustre-MDT0001 complete [11219.903345] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11222.129798] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:24:24 (1788841464) [11230.242638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11230.825615] LustreError: 379528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.865452] LustreError: 379528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11230.951355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11236.325306] LustreError: 379529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11236.360161] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11241.442702] LustreError: 379528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11245.370635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11245.670331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11249.801111] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11258.583668] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11265.698848] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11275.465298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11276.036106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11284.625358] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11293.776342] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11301.983424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11302.113743] Lustre: lustre-OST0001: new disk, initializing [11302.116447] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11302.121506] Lustre: 382368:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11302.181610] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11307.654094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11307.660664] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11307.764869] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11308.871643] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11318.911865] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11323.364738] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11323.374553] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11323.375041] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11323.375048] Lustre: Skipped 1 previous similar message [11323.380913] Lustre: Skipped 1 previous similar message [11328.498537] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11328.503030] Lustre: Skipped 3 previous similar messages [11328.859717] Lustre: server umount lustre-OST0001 complete [11331.056833] LustreError: 382668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.721599] LustreError: 381349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.739480] LustreError: 381349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11346.918816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11346.926455] LustreError: Skipped 1 previous similar message [11346.930505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11346.957125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11348.969594] LustreError: 381351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11348.998143] LustreError: 381351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11351.623834] Lustre: server umount lustre-OST0000 complete [11358.177030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11358.190691] LustreError: Skipped 1 previous similar message [11358.204852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11358.226369] Lustre: Skipped 1 previous similar message [11362.110969] Lustre: server umount lustre-MDT0000 complete [11366.445734] LustreError: 379509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841610 with bad export cookie 4211604404158472093 [11366.451610] LustreError: 379509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11366.455555] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11372.896458] Lustre: server umount lustre-MDT0001 complete [11384.536521] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11387.357256] Key type lgssc unregistered [11387.648146] LNet: 384354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11387.661334] LNetError: 384354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11388.714650] LNet: Removed LNI 192.168.202.139@tcp [11389.527071] Key type .llcrypt unregistered [11389.529775] Key type ._llcrypt unregistered [11453.343871] Key type ._llcrypt registered [11453.347319] Key type .llcrypt registered [11453.487794] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [11455.974788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11456.122498] alg: No test for adler32 (adler32-zlib) [11457.531245] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [11458.083736] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [11459.984647] Key type lgssc registered [11462.055684] Lustre: Echo OBD driver; http://www.lustre.org/ [11473.037696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11473.092090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11474.493591] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11474.520337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11474.711218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11480.039504] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11489.357075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11489.456275] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11489.481954] Lustre: 387153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11489.687745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11494.745350] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11503.274450] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11508.810551] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11511.781534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11511.786857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11511.794274] Lustre: Skipped 1 previous similar message [11516.927504] Lustre: server umount lustre-MDT0000 complete [11520.580035] LustreError: 387146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841764 with bad export cookie 11642354690647127144 [11520.598745] LustreError: 387146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11520.600092] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11521.013600] Lustre: server umount lustre-MDT0001 complete [11531.159143] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:29:33 (1788841773) [11538.638653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11539.092213] LustreError: 389663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11539.129063] LustreError: 389663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11539.146947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11543.753946] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11544.547048] LustreError: 389664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11549.667986] LustreError: 389663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11552.354095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11552.730586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11557.372760] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11565.908562] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11574.345522] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11583.042644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11583.301030] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11583.558771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11592.463252] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11601.227832] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11611.267525] Lustre: Permanently deactivating lustre-OST0000 [11611.282792] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11618.919316] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11621.450780] LustreError: 392652:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11621.457742] LustreError: 392652:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11621.461154] LustreError: 392652:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11624.964805] LustreError: 392702:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11624.972089] LustreError: 392702:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11633.288037] Lustre: server umount lustre-OST0000 complete [11636.515023] LustreError: 392951:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11636.523092] LustreError: 392951:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11639.776635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11639.790595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11639.803833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11639.806543] Lustre: Skipped 1 previous similar message [11642.336657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11642.337150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11647.458453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11647.465544] Lustre: Skipped 2 previous similar messages [11652.577607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11652.585641] Lustre: Skipped 1 previous similar message [11653.087550] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11653.485160] Lustre: server umount lustre-MDT0000 complete [11657.699206] LustreError: 391677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11657.717040] LustreError: 391677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11658.378750] LustreError: 389643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788841902 with bad export cookie 11642354690647127732 [11658.381476] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11658.392212] LustreError: 389643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11668.211481] Lustre: server umount lustre-MDT0001 complete [11678.846601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11684.462895] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11687.685603] LustreError: 394114:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11695.693956] Lustre: 394453:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.139@tcp [11697.908578] Lustre: server umount MGS complete [11712.216465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11712.952492] LustreError: 395177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11712.997448] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11713.006685] Lustre: Skipped 1 previous similar message [11713.164206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11719.531433] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11723.755433] LustreError: 395178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11723.782400] LustreError: 395178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11729.333574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11729.604859] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11729.688759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11734.574303] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11741.811475] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11747.745264] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11749.918237] Lustre: Permanently reactivating lustre-OST0000 [11749.929752] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11749.932835] Lustre: Skipped 2 previous similar messages [11778.527844] LustreError: 396965:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11786.610710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11786.945833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11793.080659] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11801.188772] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11808.739421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11808.744963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11808.757707] Lustre: Skipped 1 previous similar message [11808.769540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11808.772314] Lustre: Skipped 1 previous similar message [11809.336624] Lustre: server umount lustre-OST0000 complete [11816.952644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11816.961846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11820.791086] Lustre: server umount lustre-MDT0000 complete [11824.099712] LustreError: 395177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11824.108760] LustreError: 395177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11825.552727] LustreError: 397853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842069 with bad export cookie 11642354690647128782 [11825.561358] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11825.571533] LustreError: 397853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11826.409458] Lustre: server umount lustre-MDT0001 complete [11839.953340] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [11843.193333] Key type lgssc unregistered [11843.515242] LNet: 399541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11843.521572] LNetError: 399541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11844.585947] LNet: Removed LNI 192.168.202.139@tcp [11845.923591] Key type .llcrypt unregistered [11845.925901] Key type ._llcrypt unregistered [11865.734474] Key type ._llcrypt registered [11865.737235] Key type .llcrypt registered [11865.828520] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [11883.768216] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [11885.163612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11885.205122] alg: No test for adler32 (adler32-zlib) [11886.426749] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [11886.680115] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [11888.337244] Key type lgssc registered [11889.872641] Lustre: Echo OBD driver; http://www.lustre.org/ [11935.423294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11935.473549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11936.815428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11936.838884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11936.923252] Lustre: lustre-MDT0000: new disk, initializing [11937.010138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11937.027069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11941.549922] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11954.063061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11954.191546] Lustre: 402697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11954.259305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11954.264413] Lustre: Skipped 1 previous similar message [11954.349366] Lustre: lustre-MDT0001: new disk, initializing [11954.459359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11954.527387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11954.553064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11959.700203] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11970.047606] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11977.772307] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11984.709743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11984.971598] Lustre: lustre-OST0000: new disk, initializing [11984.978512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11984.990714] Lustre: 404669:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11985.083444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11991.061887] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [11992.148941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11992.157424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11992.255873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11999.306380] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12005.348832] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12005.610901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12011.683299] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12011.974916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12018.147314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12018.151384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12018.170519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12018.208254] Lustre: Skipped 1 previous similar message [12020.708919] Lustre: server umount lustre-OST0000 complete [12026.337729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12026.350867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12026.375232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12026.394940] Lustre: Skipped 1 previous similar message [12028.385547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12028.404190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12032.330737] Lustre: server umount lustre-MDT0000 complete [12037.009716] LustreError: 402690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842281 with bad export cookie 819617296262651853 [12037.044653] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12037.373980] Lustre: server umount lustre-MDT0001 complete [12047.148126] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12049.275916] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:38:11 (1788842291) [12057.439084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12057.864629] LustreError: 407126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12057.881938] LustreError: 407126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12057.963052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12062.730308] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12063.201269] LustreError: 407127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12068.324035] LustreError: 407126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12071.986447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12072.596991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12078.332617] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12086.862851] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12093.284818] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12101.110246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12101.521217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12109.606115] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12118.826760] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12121.467989] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12124.762397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12132.331859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12132.338429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12132.353413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12132.378151] Lustre: Skipped 2 previous similar messages [12137.440914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12137.459434] Lustre: Skipped 1 previous similar message [12138.106439] Lustre: server umount lustre-OST0000 complete [12144.098958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12144.117073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12144.133049] Lustre: Skipped 1 previous similar message [12147.700102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12147.715958] Lustre: Skipped 2 previous similar messages [12157.408163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12157.632443] Lustre: server umount lustre-MDT0000 complete [12157.926108] LustreError: 407126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12157.953959] LustreError: 407126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12161.584443] LustreError: 407109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842405 with bad export cookie 819617296262652735 [12161.590703] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12161.594223] LustreError: 407109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12161.917789] Lustre: server umount lustre-MDT0001 complete [12174.090144] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [12177.022350] Key type lgssc unregistered [12177.314764] LNet: 410852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12177.321804] LNetError: 410852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12177.337476] LNet: Removed LNI 192.168.202.139@tcp [12178.095439] Key type .llcrypt unregistered [12178.097791] Key type ._llcrypt unregistered [12201.046320] Key type ._llcrypt registered [12201.048930] Key type .llcrypt registered [12201.161186] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [12201.470674] Key type .llcrypt unregistered [12201.473278] Key type ._llcrypt unregistered [12218.107524] Key type ._llcrypt registered [12218.109503] Key type .llcrypt registered [12218.194943] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12219.113094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12219.127569] alg: No test for adler32 (adler32-zlib) [12220.244684] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [12220.455572] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [12222.087596] Key type lgssc registered [12222.966747] Lustre: Echo OBD driver; http://www.lustre.org/ [12229.889410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12229.907276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12231.313319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12235.499385] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12240.865233] LustreError: 413059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12240.891377] LustreError: 413059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12244.397657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12244.813253] LustreError: 413058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12244.841523] LustreError: 413058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12244.851877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12250.099400] LustreError: 413059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12250.121089] LustreError: 413059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12251.194028] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12255.205135] LustreError: 413058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12255.218520] LustreError: 413058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12256.247435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12258.489265] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12261.350307] LustreError: 413059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12261.371690] LustreError: 413059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12265.355509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12265.843286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12270.558463] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12280.633758] Lustre: server umount lustre-OST0000 complete [12285.932372] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12285.943218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12285.964510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12291.553761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12291.558171] Lustre: Skipped 3 previous similar messages [12294.704815] Lustre: server umount lustre-MDT0000 complete [12296.676374] LustreError: 413058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12296.697423] LustreError: 413058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12297.695923] Lustre: 412370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788842525/real 1788842525] req@00000000b394d3dd x1875737275556864/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788842541 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12297.712054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12297.730016] Lustre: Skipped 1 previous similar message [12298.406569] LustreError: 413039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842542 with bad export cookie 2431678959011362496 [12298.410694] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12298.417742] LustreError: 413039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12299.148821] Lustre: server umount lustre-MDT0001 complete [12309.975374] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [12312.349254] Key type lgssc unregistered [12312.608437] LNet: 416206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12312.621298] LNetError: 416206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12312.648470] LNet: Removed LNI 192.168.202.139@tcp [12313.639679] Key type .llcrypt unregistered [12313.643516] Key type ._llcrypt unregistered [12336.343710] Key type ._llcrypt registered [12336.345959] Key type .llcrypt registered [12336.430206] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12337.401615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12337.665086] alg: No test for adler32 (adler32-zlib) [12338.640742] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [12338.924140] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [12340.647593] Key type lgssc registered [12341.792289] Lustre: Echo OBD driver; http://www.lustre.org/ [12349.637166] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12349.655388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12365.215612] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12370.335460] LustreError: 417591:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.139@tcp: failed processing log, type 4: rc = -110 [12397.090864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12402.572867] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12409.572843] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12416.545691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12417.103831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12417.157389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12421.958836] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12422.121586] LustreError: 417616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12422.159750] LustreError: 417616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12427.259905] LustreError: 417617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12431.295549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12431.395530] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12431.402423] Lustre: Skipped 1 previous similar message [12431.628717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12436.416321] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12444.918795] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12451.736429] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12461.377454] Lustre: server umount lustre-OST0000 complete [12467.198941] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12467.215386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12467.232808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12472.801190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12472.805629] Lustre: Skipped 2 previous similar messages [12474.992272] Lustre: server umount lustre-MDT0000 complete [12477.928725] LustreError: 418632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12477.946865] LustreError: 418632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12478.625624] Lustre: 416930:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788842706/real 1788842706] req@000000005fe60896 x1875737400341888/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788842722 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12478.667284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12478.689463] Lustre: Skipped 1 previous similar message [12479.077476] LustreError: 417598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842723 with bad export cookie 17830824950785417435 [12479.093444] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12479.386334] Lustre: server umount lustre-MDT0001 complete [12490.596890] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [12493.569753] Key type lgssc unregistered [12493.995677] LNet: 421159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12494.002596] LNetError: 421159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12494.029213] LNet: Removed LNI 192.168.202.139@tcp [12494.951301] Key type .llcrypt unregistered [12494.952980] Key type ._llcrypt unregistered [12523.413662] Key type ._llcrypt registered [12523.417746] Key type .llcrypt registered [12523.565228] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12525.034135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12525.055491] alg: No test for adler32 (adler32-zlib) [12526.188686] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [12526.554618] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [12528.282625] Key type lgssc registered [12529.699641] Lustre: Echo OBD driver; http://www.lustre.org/ [12538.328900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12538.370766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12539.946992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12544.797268] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12550.112133] LustreError: 422574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12550.133539] LustreError: 422574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12555.025635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12555.231836] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12555.374787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12561.624391] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12571.424615] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12578.905424] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12587.048149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12587.455559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12595.482192] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12603.944827] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12609.970401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12615.146670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12615.151925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12615.160232] Lustre: Skipped 2 previous similar messages [12615.178661] Lustre: Skipped 2 previous similar messages [12616.202703] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.39@tcp (stopping) [12619.796720] Lustre: server umount lustre-MDT0000 complete [12620.259948] LustreError: 422573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.277965] LustreError: 422573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12621.295685] LustreError: 422570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12622.475438] LustreError: 422554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.39@tcp arrived at 1788842866 with bad export cookie 7929493172142372748 [12624.438666] LustreError: 422555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788842868 with bad export cookie 7929493172142372083 [12624.460512] LustreError: 422555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12624.473396] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12625.378130] LustreError: 423514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12625.385044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12625.395842] LustreError: 423514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12625.435174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12629.411211] LustreError: 422574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12629.451623] LustreError: 422574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12630.770492] Lustre: server umount lustre-OST0000 complete [12637.399546] Lustre: server umount lustre-MDT0001 complete [12647.807215] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [12650.722044] Key type lgssc unregistered [12650.943220] LNet: 426259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12650.953228] LNetError: 426259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12652.014605] LNet: Removed LNI 192.168.202.139@tcp [12653.040282] Key type .llcrypt unregistered [12653.042942] Key type ._llcrypt unregistered [12678.054305] Key type ._llcrypt registered [12678.062814] Key type .llcrypt registered [12678.167935] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12680.155734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12680.211147] alg: No test for adler32 (adler32-zlib) [12681.300682] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [12681.603309] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [12683.320360] Key type lgssc registered [12684.500893] Lustre: Echo OBD driver; http://www.lustre.org/ [12692.824535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12692.837443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12694.327990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12699.055988] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12704.747472] LustreError: 427672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12704.763455] LustreError: 427672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12707.623896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12708.017859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12712.873409] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12719.881358] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12725.237987] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12732.307908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12732.783833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12738.948892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12741.010912] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12749.232808] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12754.400370] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12754.409085] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12754.418487] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12754.444996] Lustre: Skipped 1 previous similar message [12757.532540] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.39@tcp (stopping) [12757.548378] Lustre: Skipped 1 previous similar message [12759.526579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12759.542370] Lustre: Skipped 1 previous similar message [12760.867613] Lustre: server umount lustre-MDT0001 complete [12762.602898] LustreError: 430151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12764.643676] LustreError: 427672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12764.657625] LustreError: 427672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12769.765582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12769.766455] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12769.766600] LustreError: 427673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12769.795597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12773.946077] Lustre: server umount lustre-OST0000 complete [12774.889260] LustreError: 427672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12778.557691] Lustre: server umount lustre-MDT0000 complete [12794.048244] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [12796.849912] Key type lgssc unregistered [12797.159669] LNet: 431355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12797.169038] LNetError: 431355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12797.196650] LNet: Removed LNI 192.168.202.139@tcp [12798.078248] Key type .llcrypt unregistered [12798.080151] Key type ._llcrypt unregistered [12825.160070] Key type ._llcrypt registered [12825.164237] Key type .llcrypt registered [12825.296772] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [12841.122936] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12842.605417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12842.688175] alg: No test for adler32 (adler32-zlib) [12843.771661] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [12844.188122] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [12845.831516] Key type lgssc registered [12847.283223] Lustre: Echo OBD driver; http://www.lustre.org/ [12888.754348] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [12909.490681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12909.560045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12910.928541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12911.009169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12911.150947] Lustre: lustre-MDT0000: new disk, initializing [12911.274472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12911.326792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12918.608755] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12940.638037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12940.935283] Lustre: lustre-OST0000: new disk, initializing [12940.937751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12940.939729] Lustre: Skipped 1 previous similar message [12940.947637] Lustre: 436389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12941.037401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12948.013444] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12948.636693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12948.642080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12948.678596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12959.693575] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12978.005447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12978.206895] Lustre: 435371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12978.359111] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12978.455539] Lustre: lustre-MDT0001: new disk, initializing [12978.641516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12978.747200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12978.770183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12983.819516] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [12994.930807] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12995.205655] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13001.011143] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13001.405978] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13008.904265] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.39@tcp (stopping) [13010.912557] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13010.926651] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13010.932686] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13010.955859] Lustre: Skipped 1 previous similar message [13010.972851] Lustre: Skipped 1 previous similar message [13013.987489] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.39@tcp (stopping) [13016.034537] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13016.052703] Lustre: Skipped 1 previous similar message [13021.151561] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13021.163090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13021.174031] Lustre: Skipped 2 previous similar messages [13021.461889] Lustre: server umount lustre-MDT0001 complete [13031.402803] LustreError: 435379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13031.427303] LustreError: 435379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13036.519952] LustreError: 435377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13036.547136] LustreError: 435377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13041.124812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13041.135844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13041.157943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13041.632477] LustreError: 435382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13046.762882] LustreError: 435383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13047.257198] Lustre: server umount lustre-OST0000 complete [13057.759682] Lustre: server umount lustre-MDT0000 complete [13073.292173] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [13077.027801] Key type lgssc unregistered [13077.366686] LNet: 440138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13077.380330] LNetError: 440138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13077.429444] LNet: Removed LNI 192.168.202.139@tcp [13078.706835] Key type .llcrypt unregistered [13078.709080] Key type ._llcrypt unregistered [13128.010161] Key type ._llcrypt registered [13128.016760] Key type .llcrypt registered [13128.170500] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [13128.504761] Key type .llcrypt unregistered [13128.513904] Key type ._llcrypt unregistered [13150.818418] Key type ._llcrypt registered [13150.820507] Key type .llcrypt registered [13150.944931] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [13152.679877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13152.757431] alg: No test for adler32 (adler32-zlib) [13154.125302] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [13154.504124] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [13156.279228] Key type lgssc registered [13157.881900] Lustre: Echo OBD driver; http://www.lustre.org/ [13193.064453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13193.104714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13194.323270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13194.368724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13194.511330] Lustre: lustre-MDT0000: new disk, initializing [13194.697212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13194.718183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13200.347231] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13212.819224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13212.887387] Lustre: 443512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13212.951303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13212.954820] Lustre: Skipped 1 previous similar message [13213.044394] Lustre: lustre-MDT0001: new disk, initializing [13213.227692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13213.303400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13213.329943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13218.284899] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13228.802919] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13236.078437] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13244.328870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13244.631398] Lustre: lustre-OST0000: new disk, initializing [13244.635218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13244.641091] Lustre: 445478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13244.743492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13251.448539] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13254.263420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13254.278416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13254.407752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13261.747091] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13269.472453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13269.480422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13269.504820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13269.989206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13270.014413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13270.015985] Lustre: Skipped 1 previous similar message [13270.038626] Lustre: Skipped 1 previous similar message [13273.851762] Lustre: server umount lustre-MDT0000 complete [13278.268111] LustreError: 443502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843522 with bad export cookie 9290778429529271243 [13278.278676] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13278.282549] LustreError: 443502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13278.844889] Lustre: server umount lustre-MDT0001 complete [13293.528479] Lustre: server umount lustre-OST0000 complete [13305.719899] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:59:08 (1788843548) [13314.712702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13315.352561] LustreError: 447508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13315.377744] LustreError: 447508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13315.439040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13320.678315] LustreError: 447509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13321.075253] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13325.792063] LustreError: 447508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13330.916030] LustreError: 447509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13331.755807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13332.364524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13337.558507] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13345.063186] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13351.257735] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13371.120839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13371.404968] Lustre: Found index 0 for lustre-OST0000, updating log [13371.588234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13379.215560] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [13385.919221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13386.990857] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13393.377497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13393.381068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13393.388843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13397.906877] Lustre: server umount lustre-OST0000 complete [13413.782548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13413.785597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13413.801836] Lustre: Skipped 2 previous similar messages [13413.826209] Lustre: Skipped 4 previous similar messages [13418.976615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13428.192918] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13428.396582] Lustre: server umount lustre-MDT0000 complete [13431.282813] LustreError: 449963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13431.316902] LustreError: 449963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13432.031562] LNet: 1 local NIs in recovery (showing 1): 192.168.202.139@tcp [13433.234888] LustreError: 447488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788843677 with bad export cookie 9290778429529272636 [13433.236415] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13433.252590] LustreError: 447488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13434.272323] Lustre: server umount lustre-MDT0001 complete [13445.745675] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:01:28 (1788843688) [14072.268575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14072.913725] LustreError: 453934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14073.021140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14078.433562] LustreError: 453935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14078.473862] LustreError: 453935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14079.394342] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14083.556721] LustreError: 453934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14089.185092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14089.600208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14094.428250] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14101.880616] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14107.831491] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14127.536729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14127.677513] Lustre: Found index 0 for lustre-OST0000, updating log [14127.906429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14133.834585] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14135.522988] LNet: 1 local NIs in recovery (showing 1): 192.168.202.139@tcp [14141.862864] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14146.024605] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14146.041637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14146.044016] Lustre: Skipped 7 previous similar messages [14150.172689] Lustre: server umount lustre-OST0000 complete [14155.047462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14155.058113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14155.060529] Lustre: Skipped 1 previous similar message [14155.079961] Lustre: Skipped 1 previous similar message [14157.090223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14157.092939] Lustre: Skipped 2 previous similar messages [14160.166178] LustreError: 453931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14160.184814] LustreError: 453931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14160.265656] Lustre: server umount lustre-MDT0000 complete [14163.935532] LustreError: 453914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844408 with bad export cookie 9290778429529273721 [14163.939105] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14163.948996] LustreError: 453914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14164.279406] Lustre: server umount lustre-MDT0001 complete [14193.964651] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [14200.031611] LNet: 1 local NIs in recovery (showing 1): 192.168.202.139@tcp [14203.197601] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [14236.357785] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:14:38 (1788844478) [14258.537605] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [14262.551538] Key type lgssc unregistered [14263.106192] LNet: 460540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14263.118389] LNetError: 460540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14391.142542] LNet: Removed LNI 192.168.202.139@tcp [14392.054680] Key type .llcrypt unregistered [14392.056850] Key type ._llcrypt unregistered [14409.623180] Key type ._llcrypt registered [14409.627450] Key type .llcrypt registered [14409.749512] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [14411.328918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14411.433196] alg: No test for adler32 (adler32-zlib) [14412.646676] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [14413.017932] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [14414.727252] Key type lgssc registered [14416.256693] Lustre: Echo OBD driver; http://www.lustre.org/ [14420.838561] LNet: Added LNI 192.168.202.139@tcp100 [8/256/0/180] [14422.174232] LNet: Added LNI 192.168.202.139@tcp101 [8/256/0/180] [14423.248718] LNet: Added LNI 192.168.202.139@tcp102 [8/256/0/180] [14424.099953] LNet: Added LNI 192.168.202.139@tcp103 [8/256/0/180] [14425.091796] LNet: Added LNI 192.168.202.139@tcp104 [8/256/0/180] [14426.053109] LNet: Added LNI 192.168.202.139@tcp105 [8/256/0/180] [14426.998474] LNet: Added LNI 192.168.202.139@tcp106 [8/256/0/180] [14427.986710] LNet: Added LNI 192.168.202.139@tcp107 [8/256/0/180] [14428.911669] LNet: Added LNI 192.168.202.139@tcp108 [8/256/0/180] [14430.033543] LNet: Added LNI 192.168.202.139@tcp109 [8/256/0/180] [14431.089320] LNet: Added LNI 192.168.202.139@tcp110 [8/256/0/180] [14432.000656] LNet: Added LNI 192.168.202.139@tcp111 [8/256/0/180] [14432.875124] LNet: Added LNI 192.168.202.139@tcp112 [8/256/0/180] [14433.977647] LNet: Added LNI 192.168.202.139@tcp113 [8/256/0/180] [14435.219726] LNet: Added LNI 192.168.202.139@tcp114 [8/256/0/180] [14436.408778] LNet: Added LNI 192.168.202.139@tcp115 [8/256/0/180] [14437.219572] LNet: Added LNI 192.168.202.139@tcp116 [8/256/0/180] [14438.125203] LNet: Added LNI 192.168.202.139@tcp117 [8/256/0/180] [14439.049523] LNet: Added LNI 192.168.202.139@tcp118 [8/256/0/180] [14440.032695] LNet: Added LNI 192.168.202.139@tcp119 [8/256/0/180] [14441.421456] LNet: Added LNI 192.168.202.139@tcp120 [8/256/0/180] [14442.683994] LNet: Added LNI 192.168.202.139@tcp121 [8/256/0/180] [14443.823773] LNet: Added LNI 192.168.202.139@tcp122 [8/256/0/180] [14444.726055] LNet: Added LNI 192.168.202.139@tcp123 [8/256/0/180] [14445.608715] LNet: Added LNI 192.168.202.139@tcp124 [8/256/0/180] [14446.440291] LNet: Added LNI 192.168.202.139@tcp125 [8/256/0/180] [14447.237284] LNet: Added LNI 192.168.202.139@tcp126 [8/256/0/180] [14448.102560] LNet: Added LNI 192.168.202.139@tcp127 [8/256/0/180] [14449.101548] LNet: Added LNI 192.168.202.139@tcp128 [8/256/0/180] [14450.036282] LNet: Added LNI 192.168.202.139@tcp129 [8/256/0/180] [14451.131780] LNet: Added LNI 192.168.202.139@tcp130 [8/256/0/180] [14452.121369] LNet: Added LNI 192.168.202.139@tcp131 [8/256/0/180] [14452.982945] LNet: Added LNI 192.168.202.139@tcp132 [8/256/0/180] [14453.909771] LNet: Added LNI 192.168.202.139@tcp133 [8/256/0/180] [14454.900582] LNet: Added LNI 192.168.202.139@tcp134 [8/256/0/180] [14455.837617] LNet: Added LNI 192.168.202.139@tcp135 [8/256/0/180] [14463.280988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14463.302239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14464.438409] Lustre: 463573:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.139@tcp: can't add NID 192.168.202.139@tcp130: rc = -75 [14464.562340] LNetError: 461029:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.139 [14464.597678] LNetError: Protocol error connecting to 192.168.202.139@tcp120 at host 192.168.202.139:988: is it running a compatible version of Lustre? [14464.626891] LNetError: Skipped 9 previous similar messages [14464.705991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14464.751352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14464.813252] Lustre: lustre-MDT0000: new disk, initializing [14464.912643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14464.928682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14465.249047] LNetError: 461028:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.139 [14465.250422] LNet: 5 local NIs in recovery (showing 5): 192.168.202.139@tcp127, 192.168.202.139@tcp128, 192.168.202.139@tcp125, 192.168.202.139@tcp129, 192.168.202.139@tcp120 [14465.258523] LNetError: 461028:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14465.258625] LNetError: Protocol error connecting to 192.168.202.139@tcp104 at host 192.168.202.139:988: is it running a compatible version of Lustre? [14467.373740] LNetError: 463604:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.139 [14467.393988] LNetError: 463604:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14467.399904] LNetError: Protocol error connecting to 192.168.202.139@tcp127 at host 192.168.202.139:988: is it running a compatible version of Lustre? [14467.412284] LNetError: Skipped 7 previous similar messages [14469.196972] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14471.458742] LNetError: 463602:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.139 [14471.477369] LNetError: 463602:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14471.496214] LNetError: Protocol error connecting to 192.168.202.139@tcp127 at host 192.168.202.139:988: is it running a compatible version of Lustre? [14479.376233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14479.556233] Lustre: 463589:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14479.645929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14479.650798] LNetError: 461030:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.139 [14479.656140] Lustre: Skipped 1 previous similar message [14479.684414] LNetError: Protocol error connecting to 192.168.202.139@tcp127 at host 192.168.202.139:988: is it running a compatible version of Lustre? [14479.755753] Lustre: lustre-MDT0001: new disk, initializing [14479.795180] Lustre: 464390:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.139@tcp: can't add NID 192.168.202.139@tcp130: rc = -75 [14479.799591] Lustre: 464390:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14479.914746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14479.962419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14479.978230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14481.306771] Lustre: 464474:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.139@tcp: can't add NID 192.168.202.139@tcp130: rc = -75 [14481.320989] Lustre: 464474:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14483.882533] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14494.172784] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14494.955286] LNetError: 463605:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.139 [14494.968265] LNetError: Protocol error connecting to 192.168.202.139@tcp126 at host 192.168.202.139:988: is it running a compatible version of Lustre? [14499.882797] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14506.575646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14506.857451] Lustre: lustre-OST0000: new disk, initializing [14506.861927] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14506.866233] Lustre: 465563:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14506.958643] Lustre: 465568:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.139@tcp: can't add NID 192.168.202.139@tcp130: rc = -75 [14506.964509] Lustre: 465568:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14507.041287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14511.329235] LNetError: 461028:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.139 [14511.343490] LNetError: 461028:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14511.352991] LNetError: Protocol error connecting to 192.168.202.139@tcp119 at host 192.168.202.139:988: is it running a compatible version of Lustre? [14511.373228] LNetError: Skipped 9 previous similar messages [14514.317570] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14516.259718] Lustre: 466020:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.139@tcp: can't add NID 192.168.202.139@tcp130: rc = -75 [14516.266161] Lustre: 466020:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14516.317270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14516.338607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14516.444858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14524.169230] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14525.727098] LNet: 1 local NIs in recovery (showing 1): 192.168.202.139@tcp127 [14532.076323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14532.086409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14532.091125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14532.100388] Lustre: Skipped 1 previous similar message [14532.165516] LustreError: Skipped 1 previous similar message [14537.193824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14537.221238] Lustre: Skipped 1 previous similar message [14542.308585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14542.314303] Lustre: Skipped 1 previous similar message [14545.890318] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14546.383962] Lustre: server umount lustre-OST0000 complete [14551.520744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14551.533200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14551.552458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14551.563727] Lustre: Skipped 1 previous similar message [14552.552637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14553.312504] LNetError: 461029:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.139 [14553.337888] LNetError: 461029:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [14553.346527] LNetError: Protocol error connecting to 192.168.202.139@tcp104 at host 192.168.202.139:988: is it running a compatible version of Lustre? [14553.363104] LNetError: Skipped 18 previous similar messages [14556.429993] Lustre: server umount lustre-MDT0000 complete [14560.939836] LustreError: 463581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788844805 with bad export cookie 16086326389339561104 [14560.951609] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14560.967361] LustreError: 463581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14561.347353] Lustre: server umount lustre-MDT0001 complete [14572.832498] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [14575.492919] Key type lgssc unregistered [14575.892300] LNet: 467475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14575.909667] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14575.923558] LNet: Removed LNI 192.168.202.139@tcp [14576.991480] LNet: Removed LNI 192.168.202.139@tcp100 [14577.004958] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14577.020164] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14577.028776] LNet: Removed LNI 192.168.202.139@tcp101 [14578.079717] LNet: Removed LNI 192.168.202.139@tcp102 [14578.084343] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14578.111361] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14578.133884] LNet: Removed LNI 192.168.202.139@tcp103 [14579.167643] LNet: Removed LNI 192.168.202.139@tcp104 [14580.191181] LNet: Removed LNI 192.168.202.139@tcp105 [14580.199309] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14580.225563] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14580.237944] LNet: Removed LNI 192.168.202.139@tcp106 [14581.279324] LNet: Removed LNI 192.168.202.139@tcp107 [14582.309380] LNet: Removed LNI 192.168.202.139@tcp108 [14583.327777] LNet: Removed LNI 192.168.202.139@tcp109 [14584.351310] LNet: Removed LNI 192.168.202.139@tcp110 [14584.353535] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14584.364056] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14584.379293] LNet: Removed LNI 192.168.202.139@tcp111 [14585.439314] LNet: Removed LNI 192.168.202.139@tcp112 [14586.463460] LNet: Removed LNI 192.168.202.139@tcp113 [14587.489760] LNet: Removed LNI 192.168.202.139@tcp114 [14588.511750] LNet: Removed LNI 192.168.202.139@tcp115 [14589.535254] LNet: Removed LNI 192.168.202.139@tcp116 [14590.559298] LNet: Removed LNI 192.168.202.139@tcp117 [14591.584116] LNet: Removed LNI 192.168.202.139@tcp118 [14592.607178] LNet: Removed LNI 192.168.202.139@tcp119 [14592.612211] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14592.622256] LNetError: 467475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14592.631516] LNet: Removed LNI 192.168.202.139@tcp120 [14593.695239] LNet: Removed LNI 192.168.202.139@tcp121 [14594.719235] LNet: Removed LNI 192.168.202.139@tcp122 [14595.743160] LNet: Removed LNI 192.168.202.139@tcp123 [14596.767171] LNet: Removed LNI 192.168.202.139@tcp124 [14597.791513] LNet: Removed LNI 192.168.202.139@tcp125 [14598.815242] LNet: Removed LNI 192.168.202.139@tcp126 [14598.828464] LNet: Removed LNI 192.168.202.139@tcp127 [14599.904420] LNet: Removed LNI 192.168.202.139@tcp128 [14600.927169] LNet: Removed LNI 192.168.202.139@tcp129 [14600.929266] LNet: Removed LNI 192.168.202.139@tcp130 [14600.944280] LNet: Removed LNI 192.168.202.139@tcp131 [14600.951717] LNet: Removed LNI 192.168.202.139@tcp132 [14600.966403] LNet: Removed LNI 192.168.202.139@tcp133 [14600.977085] LNet: Removed LNI 192.168.202.139@tcp134 [14600.990865] LNet: Removed LNI 192.168.202.139@tcp135 [14601.823495] Key type .llcrypt unregistered [14601.825305] Key type ._llcrypt unregistered [14628.202949] Key type ._llcrypt registered [14628.206708] Key type .llcrypt registered [14628.374759] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [14643.504733] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [14644.657811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14644.864537] alg: No test for adler32 (adler32-zlib) [14646.103924] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [14646.377878] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [14648.071266] Key type lgssc registered [14649.561557] Lustre: Echo OBD driver; http://www.lustre.org/ [14694.605824] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:22:16 (1788844936) [14717.375231] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [14720.581199] Key type lgssc unregistered [14720.842555] LNet: 471561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14720.849820] LNetError: 471561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14720.860423] LNet: Removed LNI 192.168.202.139@tcp [14721.689317] Key type .llcrypt unregistered [14721.694455] Key type ._llcrypt unregistered [14741.239802] Key type ._llcrypt registered [14741.243362] Key type .llcrypt registered [14741.419621] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [14742.608504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14742.880181] alg: No test for adler32 (adler32-zlib) [14744.057671] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [14744.324969] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [14745.999461] Key type lgssc registered [14747.418306] Lustre: Echo OBD driver; http://www.lustre.org/ [14759.016263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14759.042764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14760.432611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14760.548685] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14760.707501] Lustre: lustre-MDT0000: new disk, initializing [14760.814737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14760.830892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14765.717668] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14776.761820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14776.972944] Lustre: 472771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14777.028776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14777.032828] Lustre: Skipped 1 previous similar message [14777.109237] Lustre: lustre-MDT0001: new disk, initializing [14777.194850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14777.234799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14777.244823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14781.628606] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14791.857986] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14797.709497] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14807.183847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14807.699394] Lustre: lustre-OST0000: new disk, initializing [14807.704108] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14807.711080] Lustre: 474735:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14807.857343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14814.844280] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [14816.846148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14816.860483] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14816.954435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14825.826389] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14831.080583] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing lnet_if_list [14837.665541] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14837.774764] LNet: Added LNI 192.168.202.139@tcp10 [8/256/0/180] [14844.749832] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14844.805285] LNet: Added LNI 192.168.202.139@tcp11 [8/256/0/180] [14851.819764] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14851.886812] LNet: Added LNI 192.168.202.139@tcp12 [8/256/0/180] [14858.913735] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14858.984120] LNet: Added LNI 192.168.202.139@tcp13 [8/256/0/180] [14865.020593] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14865.048580] LNet: Added LNI 192.168.202.139@tcp14 [8/256/0/180] [14869.231260] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14869.251865] LNet: Added LNI 192.168.202.139@tcp15 [8/256/0/180] [14874.179610] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14874.210223] LNet: Added LNI 192.168.202.139@tcp16 [8/256/0/180] [14879.425408] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14879.447687] LNet: Added LNI 192.168.202.139@tcp17 [8/256/0/180] [14885.908131] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14885.978447] LNet: Added LNI 192.168.202.139@tcp18 [8/256/0/180] [14891.662639] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14891.688669] LNet: Added LNI 192.168.202.139@tcp19 [8/256/0/180] [14897.476305] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14897.497348] LNet: Added LNI 192.168.202.139@tcp20 [8/256/0/180] [14903.127529] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14903.156979] LNet: Added LNI 192.168.202.139@tcp21 [8/256/0/180] [14909.236354] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14909.293695] LNet: Added LNI 192.168.202.139@tcp22 [8/256/0/180] [14914.935628] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14914.980632] LNet: Added LNI 192.168.202.139@tcp23 [8/256/0/180] [14920.736808] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14920.761857] LNet: Added LNI 192.168.202.139@tcp24 [8/256/0/180] [14927.023425] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14927.056419] LNet: Added LNI 192.168.202.139@tcp25 [8/256/0/180] [14933.689333] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14933.719531] LNet: Added LNI 192.168.202.139@tcp26 [8/256/0/180] [14939.007130] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14939.032910] LNet: Added LNI 192.168.202.139@tcp27 [8/256/0/180] [14945.529668] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14945.595829] LNet: Added LNI 192.168.202.139@tcp28 [8/256/0/180] [14952.040912] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14952.067358] LNet: Added LNI 192.168.202.139@tcp29 [8/256/0/180] [14957.630583] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14957.685758] LNet: Added LNI 192.168.202.139@tcp30 [8/256/0/180] [14963.977763] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14963.998973] LNet: Added LNI 192.168.202.139@tcp31 [8/256/0/180] [14970.734971] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14970.757987] LNet: Added LNI 192.168.202.139@tcp32 [8/256/0/180] [14977.229439] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14977.253724] LNet: Added LNI 192.168.202.139@tcp33 [8/256/0/180] [14983.669720] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14983.691431] LNet: Added LNI 192.168.202.139@tcp34 [8/256/0/180] [14991.148371] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14991.220656] LNet: Added LNI 192.168.202.139@tcp35 [8/256/0/180] [15000.107788] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15000.145780] LNet: Added LNI 192.168.202.139@tcp36 [8/256/0/180] [15009.292516] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15009.329406] LNet: Added LNI 192.168.202.139@tcp37 [8/256/0/180] [15019.832914] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15019.912266] LNet: Added LNI 192.168.202.139@tcp38 [8/256/0/180] [15029.468896] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15029.504708] LNet: Added LNI 192.168.202.139@tcp39 [8/256/0/180] [15038.918732] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15038.981171] LNet: Added LNI 192.168.202.139@tcp40 [8/256/0/180] [15050.097582] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15050.151330] LNet: Added LNI 192.168.202.139@tcp41 [8/256/0/180] [15057.908391] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15057.933776] LNet: Added LNI 192.168.202.139@tcp42 [8/256/0/180] [15067.350969] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15067.398672] LNet: Added LNI 192.168.202.139@tcp43 [8/256/0/180] [15075.956875] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15075.978474] LNet: Added LNI 192.168.202.139@tcp44 [8/256/0/180] [15085.175482] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15085.239714] LNet: Added LNI 192.168.202.139@tcp45 [8/256/0/180] [15094.366240] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15094.418632] LNet: Added LNI 192.168.202.139@tcp46 [8/256/0/180] [15103.275100] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15103.337409] LNet: Added LNI 192.168.202.139@tcp47 [8/256/0/180] [15112.146752] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15112.176890] LNet: Added LNI 192.168.202.139@tcp48 [8/256/0/180] [15120.076549] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15120.110941] LNet: Added LNI 192.168.202.139@tcp49 [8/256/0/180] [15128.998467] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15129.028291] LNet: Added LNI 192.168.202.139@tcp50 [8/256/0/180] [15155.172437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15155.186534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15155.213489] Lustre: Skipped 1 previous similar message [15159.044754] Lustre: server umount lustre-MDT0000 complete [15164.137805] LustreError: 472762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845408 with bad export cookie 17192967828410175904 [15164.141915] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15164.144755] LustreError: 472762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15164.530937] Lustre: server umount lustre-MDT0001 complete [15181.735264] Lustre: 472043:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845409/real 1788845409] req@00000000f77470cc x1875739922316928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15181.787714] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15181.793893] Lustre: Skipped 1 previous similar message [15183.231833] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [15184.863117] Lustre: 472040:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845413/real 1788845413] req@00000000ae4fbde1 x1875739922317056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845429 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15185.891493] Lustre: 472041:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845414/real 1788845414] req@0000000097520245 x1875739922317184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845430 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15189.920074] Lustre: 472043:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845418/real 1788845418] req@0000000010b8d6ab x1875739922317440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15192.323371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15195.103151] Lustre: 472042:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788845423/real 1788845423] req@000000005af9f118 x1875739922317568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788845439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15209.442857] LustreError: 472039:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000087301972 x1875739922318336/t0(0) o250->MGC192.168.202.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15209.853078] LustreError: 474729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15209.875032] LustreError: 474729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15209.930676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15210.977065] LustreError: 474730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15211.017890] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15212.004143] LustreError: 474729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15215.451814] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15217.129773] LustreError: 474730:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15222.248979] LustreError: 474729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15225.928803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15226.398239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15227.445817] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15231.422761] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15238.289676] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15244.104157] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15261.163329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15261.178527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15261.205854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15261.218064] Lustre: Skipped 2 previous similar messages [15262.179664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15263.213200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15263.222766] Lustre: Skipped 2 previous similar messages [15266.811946] Lustre: server umount lustre-OST0000 complete [15272.420823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15272.439415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15272.458581] Lustre: Skipped 1 previous similar message [15272.473808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15276.961839] Lustre: server umount lustre-MDT0000 complete [15278.566822] LustreError: 500060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15278.585968] LustreError: 500060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15281.903341] LustreError: 472761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845526 with bad export cookie 17192967828410178578 [15281.908444] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15281.914233] LustreError: 472761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15282.219490] Lustre: server umount lustre-MDT0001 complete [15294.627170] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [15297.482594] Key type lgssc unregistered [15297.904580] LNet: 501790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15297.910561] LNetError: 501790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15298.975281] LNet: Removed LNI 192.168.202.139@tcp [15298.979042] LNetError: 501790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15298.995315] LNet: Removed LNI 192.168.202.139@tcp10 [15299.000728] LNet: Removed LNI 192.168.202.139@tcp11 [15299.003906] LNet: Removed LNI 192.168.202.139@tcp12 [15299.009727] LNet: Removed LNI 192.168.202.139@tcp13 [15299.012199] LNet: Removed LNI 192.168.202.139@tcp14 [15299.015792] LNet: Removed LNI 192.168.202.139@tcp15 [15299.020055] LNet: Removed LNI 192.168.202.139@tcp16 [15299.025579] LNet: Removed LNI 192.168.202.139@tcp17 [15299.028460] LNet: Removed LNI 192.168.202.139@tcp18 [15299.031336] LNet: Removed LNI 192.168.202.139@tcp19 [15299.034406] LNet: Removed LNI 192.168.202.139@tcp20 [15299.036611] LNet: Removed LNI 192.168.202.139@tcp21 [15299.039365] LNet: Removed LNI 192.168.202.139@tcp22 [15299.044805] LNet: Removed LNI 192.168.202.139@tcp23 [15299.049191] LNet: Removed LNI 192.168.202.139@tcp24 [15299.052369] LNet: Removed LNI 192.168.202.139@tcp25 [15299.056325] LNet: Removed LNI 192.168.202.139@tcp26 [15299.059317] LNet: Removed LNI 192.168.202.139@tcp27 [15299.061857] LNet: Removed LNI 192.168.202.139@tcp28 [15299.064389] LNet: Removed LNI 192.168.202.139@tcp29 [15299.067340] LNet: Removed LNI 192.168.202.139@tcp30 [15299.070359] LNet: Removed LNI 192.168.202.139@tcp31 [15299.073291] LNet: Removed LNI 192.168.202.139@tcp32 [15299.076658] LNet: Removed LNI 192.168.202.139@tcp33 [15299.079507] LNet: Removed LNI 192.168.202.139@tcp34 [15299.083548] LNet: Removed LNI 192.168.202.139@tcp35 [15299.086752] LNet: Removed LNI 192.168.202.139@tcp36 [15299.091710] LNet: Removed LNI 192.168.202.139@tcp37 [15299.093756] LNet: Removed LNI 192.168.202.139@tcp38 [15299.097830] LNet: Removed LNI 192.168.202.139@tcp39 [15299.103464] LNet: Removed LNI 192.168.202.139@tcp40 [15299.107748] LNet: Removed LNI 192.168.202.139@tcp41 [15299.110740] LNet: Removed LNI 192.168.202.139@tcp42 [15299.114411] LNet: Removed LNI 192.168.202.139@tcp43 [15299.117691] LNet: Removed LNI 192.168.202.139@tcp44 [15299.120688] LNet: Removed LNI 192.168.202.139@tcp45 [15299.123753] LNet: Removed LNI 192.168.202.139@tcp46 [15299.126259] LNet: Removed LNI 192.168.202.139@tcp47 [15299.128392] LNet: Removed LNI 192.168.202.139@tcp48 [15299.131355] LNet: Removed LNI 192.168.202.139@tcp49 [15299.142838] LNet: Removed LNI 192.168.202.139@tcp50 [15300.281651] Key type .llcrypt unregistered [15300.283929] Key type ._llcrypt unregistered [15325.369258] Key type ._llcrypt registered [15325.371142] Key type .llcrypt registered [15325.447097] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [15345.254309] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [15347.743093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15347.872741] alg: No test for adler32 (adler32-zlib) [15349.092289] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [15349.387410] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [15351.223206] Key type lgssc registered [15352.735064] Lustre: Echo OBD driver; http://www.lustre.org/ [15385.258880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15385.290663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15386.649376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15386.716373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15386.844216] Lustre: lustre-MDT0000: new disk, initializing [15386.958558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15386.979572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15391.870307] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15401.650554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15401.798716] Lustre: 504987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15401.851509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15401.854448] Lustre: Skipped 1 previous similar message [15401.940405] Lustre: lustre-MDT0001: new disk, initializing [15402.042207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15402.117059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15402.129801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15406.248701] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15415.658842] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15420.843207] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15427.721485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15427.862699] Lustre: *** cfs_fail_loc=90f, val=0*** [15427.930743] Lustre: lustre-OST0000: new disk, initializing [15427.934335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15427.940112] Lustre: 507001:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15428.048950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15434.593339] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15436.415356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15436.436888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15436.602055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15444.083339] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15457.253736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15457.259736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15457.261718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15457.266943] LustreError: Skipped 1 previous similar message [15457.302295] Lustre: Skipped 1 previous similar message [15458.840048] Lustre: server umount lustre-OST0000 complete [15463.393457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15463.399331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15463.417352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15463.425419] Lustre: Skipped 1 previous similar message [15467.488605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15467.490195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15467.521503] Lustre: Skipped 1 previous similar message [15469.249190] Lustre: server umount lustre-MDT0000 complete [15473.239459] LustreError: 505790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845717 with bad export cookie 6276036990759234529 [15473.242748] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15473.250477] LustreError: 505790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15473.620895] Lustre: server umount lustre-MDT0001 complete [15487.332579] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [15489.190695] Key type lgssc unregistered [15489.601110] LNet: 508906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15489.615537] LNetError: 508906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15489.651140] LNet: Removed LNI 192.168.202.139@tcp [15490.732134] Key type .llcrypt unregistered [15490.734916] Key type ._llcrypt unregistered [15510.057093] Key type ._llcrypt registered [15510.058965] Key type .llcrypt registered [15510.202773] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [15511.688288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15511.730782] alg: No test for adler32 (adler32-zlib) [15512.822551] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [15513.045259] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [15514.695230] Key type lgssc registered [15515.938036] Lustre: Echo OBD driver; http://www.lustre.org/ [15523.125380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15523.156581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15524.710248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15529.838499] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15534.054795] LustreError: 510175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15534.065475] LustreError: 510175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15538.837764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15539.154738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15543.421432] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15551.187039] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15556.993606] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15563.822661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15564.270726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15570.776226] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15576.999658] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15585.587710] loop0: detected capacity change from 0 to 204800000 [15585.820964] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15585.833788] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15585.841657] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15591.925448] loop0: detected capacity change from 0 to 204800000 [15592.060538] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15592.072695] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15592.085210] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15596.912101] loop0: detected capacity change from 0 to 204800000 [15601.567669] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15601.695973] Lustre: fs15246-OST0000: new disk, initializing [15601.700922] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15601.709583] Lustre: 513766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15601.773796] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15607.469701] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15617.266214] loop1: detected capacity change from 0 to 204800000 [15623.083858] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15623.304243] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15623.393608] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15623.544972] Lustre: fs15246-MDT0000: new disk, initializing [15623.772464] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15623.807556] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15624.018510] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15625.059490] Lustre: 514887:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000010b8d6ab x1875740728536960/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788845885 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15628.783234] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15634.148641] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15634.161457] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15634.196857] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15650.796553] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15650.814566] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15655.906521] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15656.812717] Lustre: server umount fs15246-MDT0000 complete [15661.049164] LustreError: 511416:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15665.535530] Lustre: server umount fs15246-OST0000 complete [15676.388573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15676.399809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15676.402981] Lustre: Skipped 1 previous similar message [15681.510419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15681.521153] Lustre: Skipped 1 previous similar message [15681.754274] Lustre: server umount lustre-OST0000 complete [15686.625715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15686.634176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15686.657421] Lustre: Skipped 2 previous similar messages [15686.677085] Lustre: Skipped 1 previous similar message [15691.631256] Lustre: server umount lustre-MDT0000 complete [15691.748129] LustreError: 511211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15691.774109] LustreError: 511211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15695.347429] LustreError: 510156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788845939 with bad export cookie 2466088411935370210 [15695.369421] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15695.384661] LustreError: 510156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15695.982918] Lustre: server umount lustre-MDT0001 complete [15706.358946] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [15708.727967] Key type lgssc unregistered [15709.010912] LNet: 517984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15709.020504] LNetError: 517984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15709.040523] LNet: Removed LNI 192.168.202.139@tcp [15709.856757] Key type .llcrypt unregistered [15709.860487] Key type ._llcrypt unregistered [15728.225726] Key type ._llcrypt registered [15728.227806] Key type .llcrypt registered [15728.348392] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [15729.476147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15729.534956] alg: No test for adler32 (adler32-zlib) [15730.788232] Lustre: Lustre: Build Version: 2.17.57_104_ga33798f [15731.088164] LNet: Added LNI 192.168.202.139@tcp [8/256/0/180] [15732.775248] Key type lgssc registered [15734.299498] Lustre: Echo OBD driver; http://www.lustre.org/ [15779.802599] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [15790.940572] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [15828.989328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15829.045517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15830.391448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15830.419138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15830.479276] Lustre: lustre-MDT0000: new disk, initializing [15830.536499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15830.554836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15834.934692] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15846.093705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15846.322959] Lustre: 522452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15846.409693] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15846.413447] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15846.416628] Lustre: Skipped 1 previous similar message [15846.499426] Lustre: lustre-MDT0001: new disk, initializing [15846.598893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15846.681112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15846.696747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15851.927471] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15861.404922] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15868.247605] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15875.402466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15875.667687] Lustre: lustre-OST0000: new disk, initializing [15875.676067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15875.683078] Lustre: 524420:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15875.766207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15881.799356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15881.813722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15881.922911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15882.711300] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_default_debug -1 all [15895.331560] Lustre: DEBUG MARKER: oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15899.786603] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15900.174723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15905.125509] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15905.371867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15907.812268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15907.814111] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15907.826890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15907.827152] Lustre: Skipped 1 previous similar message [15912.939455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15912.943925] Lustre: Skipped 2 previous similar messages [15913.426421] Lustre: server umount lustre-OST0000 complete [15918.049675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15918.056569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15918.080257] Lustre: Skipped 1 previous similar message [15923.170635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15923.192339] Lustre: Skipped 2 previous similar messages [15923.552875] Lustre: server umount lustre-MDT0000 complete [15926.740735] LustreError: 522443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788846170 with bad export cookie 5186557837750617370 [15926.749627] LustreError: MGC192.168.202.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15926.761097] LustreError: 522443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15927.164227] Lustre: server umount lustre-MDT0001 complete [15954.648364] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing set_hostid [15964.027402] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing load_modules_local [15994.152942] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15699 sec ========= 01:43:56 (1788846236) [15996.091129] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:43:58 (1788846238) === [15999.783284] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:44:02 (1788846242) === [16044.213033] Lustre: DEBUG MARKER: oleg239-server.virtnet: executing unload_modules_local [16047.874825] Key type lgssc unregistered [16048.171185] LNet: 531066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16048.184499] LNetError: 531066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16049.254231] LNet: Removed LNI 192.168.202.139@tcp [16050.493781] Key type .llcrypt unregistered [16050.496218] Key type ._llcrypt unregistered