[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491823762 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003264] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008411] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011007] pid_max: default: 32768 minimum: 301 [ 0.013028] LSM: Security Framework initializing [ 0.014052] Yama: becoming mindful. [ 0.015036] SELinux: Initializing. [ 0.016063] *** VALIDATE selinux *** [ 0.024179] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028783] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030120] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032039] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034004] *** VALIDATE tmpfs *** [ 0.035453] *** VALIDATE proc *** [ 0.037089] *** VALIDATE cgroup *** [ 0.038009] *** VALIDATE cgroup2 *** [ 0.040102] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043028] Spectre V2 : User space: Vulnerable [ 0.044007] Speculative Store Bypass: Vulnerable [ 0.046614] debug: unmapping init [mem 0xffffffff9aa59000-0xffffffff9aa60fff] [ 0.048203] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049558] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050022] ... version: 2 [ 0.051012] ... bit width: 48 [ 0.051966] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053014] ... max period: 00007fffffffffff [ 0.054011] ... fixed-purpose events: 3 [ 0.054917] ... event mask: 000000070000000f [ 0.055256] rcu: Hierarchical SRCU implementation. [ 0.057351] smp: Bringing up secondary CPUs ... [ 0.058545] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.062063] smp: Brought up 1 node, 4 CPUs [ 0.064011] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.133727] node 0 deferred pages initialised in 66ms [ 0.135090] devtmpfs: initialized [ 0.136228] x86/mm: Memory block size: 128MB [ 0.138438] gcov: version magic: 0x41383552 [ 0.140188] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142268] pinctrl core: initialized pinctrl subsystem [ 0.143203] [ 0.143777] ************************************************************* [ 0.144018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145015] ** ** [ 0.146012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147011] ** ** [ 0.148013] ** This means that this kernel is built to expose internal ** [ 0.149015] ** IOMMU data structures, which may compromise security on ** [ 0.150014] ** your system. ** [ 0.151030] ** ** [ 0.152018] ** If you see this message and you are not debugging the ** [ 0.153014] ** kernel, report this immediately to your vendor! ** [ 0.154016] ** ** [ 0.155014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156012] ************************************************************* [ 0.157631] NET: Registered protocol family 16 [ 0.158465] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.159064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.160066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161479] cpuidle: using governor menu [ 0.162000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.165432] PCI: Using configuration type 1 for base access [ 0.167097] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.173051] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.174077] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176163] cryptd: max_cpu_qlen set to 1000 [ 0.178190] ACPI: Added _OSI(Module Device) [ 0.179011] ACPI: Added _OSI(Processor Device) [ 0.180008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.182012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.186498] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.191604] ACPI: Interpreter enabled [ 0.193060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195014] ACPI: Using IOAPIC for interrupt routing [ 0.196137] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.200526] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.210688] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.212031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.214013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216111] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.220085] acpiphp: Slot [2] registered [ 0.221083] acpiphp: Slot [5] registered [ 0.222063] acpiphp: Slot [6] registered [ 0.223009] acpiphp: Slot [7] registered [ 0.224047] acpiphp: Slot [8] registered [ 0.225064] acpiphp: Slot [9] registered [ 0.226123] acpiphp: Slot [10] registered [ 0.227091] acpiphp: Slot [3] registered [ 0.227916] acpiphp: Slot [4] registered [ 0.228076] acpiphp: Slot [11] registered [ 0.228868] acpiphp: Slot [12] registered [ 0.229051] acpiphp: Slot [13] registered [ 0.230107] acpiphp: Slot [14] registered [ 0.231055] acpiphp: Slot [15] registered [ 0.231958] acpiphp: Slot [16] registered [ 0.233053] acpiphp: Slot [17] registered [ 0.233961] acpiphp: Slot [18] registered [ 0.235054] acpiphp: Slot [19] registered [ 0.235978] acpiphp: Slot [20] registered [ 0.237073] acpiphp: Slot [21] registered [ 0.238064] acpiphp: Slot [22] registered [ 0.239062] acpiphp: Slot [23] registered [ 0.240082] acpiphp: Slot [24] registered [ 0.242075] acpiphp: Slot [25] registered [ 0.243081] acpiphp: Slot [26] registered [ 0.244097] acpiphp: Slot [27] registered [ 0.245072] acpiphp: Slot [28] registered [ 0.245994] acpiphp: Slot [29] registered [ 0.247054] acpiphp: Slot [30] registered [ 0.248066] acpiphp: Slot [31] registered [ 0.248973] PCI host bridge to bus 0000:00 [ 0.250017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.251013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.253016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.254015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.256020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.257016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.259130] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.260850] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.262995] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.270015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.274042] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.276016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.277013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.279016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.280443] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.282601] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.284031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.286837] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.291013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.303882] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.307014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.311298] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.316020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.320014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.333013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.342302] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.349017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.357016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.371015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.380301] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.389015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.395018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.409034] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.417587] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.423018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.430017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.447027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.457348] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.466017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.477018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.504020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.513450] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.521020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.529021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.546017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.559241] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.561382] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.563357] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.565459] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.567186] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.573035] iommu: Default domain type: Passthrough [ 0.574417] SCSI subsystem initialized [ 0.576132] ACPI: bus type USB registered [ 0.577110] usbcore: registered new interface driver usbfs [ 0.579045] usbcore: registered new interface driver hub [ 0.581110] usbcore: registered new device driver usb [ 0.583150] pps_core: LinuxPPS API ver. 1 registered [ 0.585011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.588061] PTP clock support registered [ 0.590119] EDAC MC: Ver: 3.0.0 [ 0.592150] PCI: Using ACPI for IRQ routing [ 0.594875] NetLabel: Initializing [ 0.596015] NetLabel: domain hash size = 128 [ 0.597010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.599084] NetLabel: unlabeled traffic allowed by default [ 0.601124] vgaarb: loaded [ 0.603264] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.605009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.614000] clocksource: Switched to clocksource kvm-clock [ 0.716773] VFS: Disk quotas dquot_6.6.0 [ 0.718339] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.720299] *** VALIDATE ramfs *** [ 0.721090] *** VALIDATE hugetlbfs *** [ 0.722094] pnp: PnP ACPI init [ 0.724596] pnp: PnP ACPI: found 6 devices [ 0.740719] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.743929] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.745934] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.748192] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.750496] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.752276] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.754979] NET: Registered protocol family 2 [ 0.757393] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.762303] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.766408] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.772309] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.775734] TCP: Hash tables configured (established 65536 bind 65536) [ 0.778763] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.781593] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.784237] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.787306] NET: Registered protocol family 1 [ 0.790768] RPC: Registered named UNIX socket transport module. [ 0.793035] RPC: Registered udp transport module. [ 0.794849] RPC: Registered tcp transport module. [ 0.796649] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.799135] NET: Registered protocol family 44 [ 0.800664] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.802697] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.804569] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.806707] PCI: CLS 0 bytes, default 64 [ 0.809188] Unpacking initramfs... [ 2.218991] debug: unmapping init [mem 0xffff96147cc54000-0xffff96147ffbffff] [ 2.222976] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.225325] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.227389] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.733196] Initialise system trusted keyrings [ 2.734455] Key type blacklist registered [ 2.736136] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.746531] zbud: loaded [ 2.749271] *** VALIDATE nfs *** [ 2.750613] *** VALIDATE nfs4 *** [ 2.752589] pstore: using deflate compression [ 2.756767] Platform Keyring initialized [ 2.849141] NET: Registered protocol family 38 [ 2.851092] Key type asymmetric registered [ 2.852532] Asymmetric key parser 'x509' registered [ 2.854253] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.857133] io scheduler mq-deadline registered [ 2.858798] io scheduler kyber registered [ 2.860502] io scheduler bfq registered [ 2.862656] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.865757] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.868806] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.871563] ACPI: Power Button [PWRF] [ 2.876838] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.883224] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.906410] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.918541] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.940140] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.967896] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.992909] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.996915] Non-volatile memory driver v1.3 [ 2.998423] Linux agpgart interface v0.103 [ 3.030220] virtio_blk virtio1: [vda] 149992 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.032796] vda: detected capacity change from 0 to 76795904 [ 3.047477] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.050331] vdb: detected capacity change from 0 to 1073741824 [ 3.071575] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.075276] vdc: detected capacity change from 0 to 2621440000 [ 3.089480] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.092470] vdd: detected capacity change from 0 to 2621440000 [ 3.110589] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.112719] vde: detected capacity change from 0 to 4294967296 [ 3.125275] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.128336] vdf: detected capacity change from 0 to 4294967296 [ 3.132684] libphy: Fixed MDIO Bus: probed [ 3.141712] usbcore: registered new interface driver usbserial_generic [ 3.143353] usbserial: USB Serial support registered for generic [ 3.144974] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.147978] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.149153] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.151196] mousedev: PS/2 mouse device common for all mice [ 3.153289] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.155490] rtc_cmos 00:05: RTC can wake from S4 [ 3.158287] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.160246] rtc_cmos 00:05: registered as rtc0 [ 3.163280] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.166690] intel_pstate: CPU model not supported [ 3.169152] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.172832] hid: raw HID events driver (C) Jiri Kosina [ 3.175262] usbcore: registered new interface driver usbhid [ 3.178688] usbhid: USB HID core driver [ 3.180350] drop_monitor: Initializing network drop monitor service [ 3.182844] Initializing XFRM netlink socket [ 3.184700] NET: Registered protocol family 10 [ 3.187733] Segment Routing with IPv6 [ 3.189287] NET: Registered protocol family 17 [ 3.191937] mpls_gso: MPLS GSO support [ 3.198477] RAS: Correctable Errors collector initialized. [ 3.200787] AVX version of gcm_enc/dec engaged. [ 3.202589] AES CTR mode by8 optimization enabled [ 3.274446] sched_clock: Marking stable (3274416289, 0)->(4194429521, -920013232) [ 3.277106] registered taskstats version 1 [ 3.279016] Loading compiled-in X.509 certificates [ 3.280629] zswap: loaded using pool lzo/zbud [ 3.299401] Key type big_key registered [ 3.315248] Key type encrypted registered [ 3.316871] ima: No TPM chip found, activating TPM-bypass! [ 3.318933] ima: Allocated hash algorithm: sha1 [ 3.320777] ima: No architecture policies found [ 3.322454] evm: Initialising EVM extended attributes: [ 3.324363] evm: security.selinux [ 3.325724] evm: security.ima [ 3.327104] evm: security.capability [ 3.328585] evm: HMAC attrs: 0x1 [ 3.330967] rtc_cmos 00:05: setting system clock to 2026-09-10 14:28:52 UTC (1789050532) [ 3.337664] debug: unmapping init [mem 0xffffffff9ba03000-0xffffffff9bbfffff] [ 3.340930] debug: unmapping init [mem 0xffffffff9a782000-0xffffffff9aa58fff] [ 3.348116] Write protecting the kernel read-only data: 28672k [ 3.351435] debug: unmapping init [mem 0xffffffff98e03000-0xffffffff98ffffff] [ 3.354381] debug: unmapping init [mem 0xffffffff99714000-0xffffffff997fffff] [ 3.386771] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.396127] systemd[1]: Detected virtualization kvm. [ 3.397514] systemd[1]: Detected architecture x86-64. [ 3.399151] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.423662] systemd[1]: No hostname configured. [ 3.424846] systemd[1]: Set hostname to . [ 3.426592] random: systemd: uninitialized urandom read (16 bytes read) [ 3.429209] systemd[1]: Initializing machine ID from random generator. [ 3.463736] random: ln: uninitialized urandom read (6 bytes read) [ 3.539394] random: systemd: uninitialized urandom read (16 bytes read) [ 3.541413] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.544979] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.548210] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.042787] device-mapper: uevent: version 1.0.3 [ 4.044447] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.619229] scsi host0: ata_piix [ 4.621984] virtio_net virtio0 ens2: renamed from eth0 [ 4.623856] scsi host1: ata_piix [ 4.632339] random: fast init done [ 4.662141] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.664967] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.941593] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.834674] random: crng init done [ 9.835693] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.996432] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.023184] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.238929] SELinux: Disabled at runtime. [ 11.297934] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.305408] systemd[1]: Detected virtualization kvm. [ 11.307158] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.731458] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.735313] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.739505] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.743503] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.746994] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.754600] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.758531] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Sys[ 11.841539] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS tems. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.152633] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.416181] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.429074] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.545582] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.557668] EDAC sbridge: Ver: 1.1.2 [ 14.026752] Key type dns_resolver registered [ 14.327119] NFS: Registering the id_resolver key type [ 14.328949] Key type id_resolver registered [ 14.330519] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg157-server login: [ 77.095219] libcfs: loading out-of-tree module taints kernel. [ 77.166136] Key type ._llcrypt registered [ 77.169542] Key type .llcrypt registered [ 77.377669] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 96.087072] hrtimer: interrupt took 2062566 ns [ 98.274569] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 99.271263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 99.285112] alg: No test for adler32 (adler32-zlib) [ 100.895975] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 101.666177] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 103.400506] Key type lgssc registered [ 105.234353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 123.802690] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.797505] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 180.751405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 180.811905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.127991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.180800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.326299] Lustre: lustre-MDT0000: new disk, initializing [ 182.433386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.469293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 188.383610] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.846125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.951797] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 201.982617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 201.987625] Lustre: Skipped 1 previous similar message [ 202.067619] Lustre: lustre-MDT0001: new disk, initializing [ 202.136703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.167914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 202.188224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 206.966622] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.585278] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.158793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.552172] Lustre: lustre-OST0000: new disk, initializing [ 224.557687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.573288] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.682791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.463110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 227.480198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 227.523494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 231.855695] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.478670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.635964] Lustre: lustre-OST0001: new disk, initializing [ 247.641323] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 247.648421] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.748811] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.363399] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.660996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 257.675305] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 257.815061] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.171345] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 277.901435] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 284.290265] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing check_logdir /tmp/testlogs/ [ 289.351447] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing yml_node [ 294.139287] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 297.186550] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 299.874698] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 302.116627] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 10 10:33:50 EDT 2026 [ 319.884962] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 321.610538] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 339.937635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.940221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 339.960421] Lustre: Skipped 3 previous similar messages [ 339.964652] Lustre: Skipped 2 previous similar messages [ 344.721771] Lustre: server umount lustre-MDT0000 complete [ 349.775191] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789050878 with bad export cookie 7410699505707976517 [ 349.777142] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 349.818907] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 350.176532] LustreError: 6520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.181241] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.193783] LustreError: 6520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 350.195421] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.195426] Lustre: Skipped 1 previous similar message [ 350.232947] Lustre: Skipped 1 previous similar message [ 354.210782] LustreError: 10414:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 354.214860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 354.231817] LustreError: 10414:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 354.274238] Lustre: Skipped 2 previous similar messages [ 356.184784] Lustre: server umount lustre-MDT0001 complete [ 360.389126] Lustre: server umount lustre-OST0000 complete [ 364.150298] Lustre: server umount lustre-OST0001 complete [ 373.522747] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 385.086437] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 422.441512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.731880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.759028] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.836689] Lustre: lustre-MDT0000: new disk, initializing [ 422.893421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.910306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 427.449827] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 437.735114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.949992] Lustre: 15078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 438.025546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 438.029631] Lustre: Skipped 1 previous similar message [ 438.113686] Lustre: lustre-MDT0001: new disk, initializing [ 438.234327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 438.279215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 438.290917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 443.122044] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 453.268432] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 460.343520] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 467.067970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 467.421614] Lustre: lustre-OST0000: new disk, initializing [ 467.436458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 467.445744] Lustre: 17048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 467.564845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.401288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 469.410255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 469.612197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 474.574284] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 485.113392] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.695502] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 490.029571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 495.221182] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 495.494988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 500.193327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 500.196158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.211996] LustreError: Skipped 1 previous similar message [ 500.216148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 500.242557] Lustre: Skipped 1 previous similar message [ 503.870589] Lustre: server umount lustre-OST0000 complete [ 509.924285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.940349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.953851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 509.957930] Lustre: Skipped 1 previous similar message [ 514.945275] Lustre: server umount lustre-MDT0000 complete [ 515.553807] LustreError: 15925:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.581154] LustreError: 15925:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 519.335536] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051048 with bad export cookie 7410699505707978050 [ 519.343986] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 519.359278] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 519.881299] Lustre: server umount lustre-MDT0001 complete [ 524.993386] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:37:33 (1789051053) [ 527.820937] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 529.999356] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:37:38 (1789051058) [ 538.116472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.770411] LustreError: 19458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.866711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.024608] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 544.227106] LustreError: 19459:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.402502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.295795] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 564.669924] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 570.064709] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 577.437024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 583.340722] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 590.623072] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 607.715601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 607.726628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.728189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 607.744922] Lustre: Skipped 2 previous similar messages [ 607.772125] Lustre: Skipped 3 previous similar messages [ 612.494633] Lustre: server umount lustre-OST0000 complete [ 617.954091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.971540] Lustre: Skipped 1 previous similar message [ 622.633819] Lustre: server umount lustre-MDT0000 complete [ 623.074407] LustreError: 20188:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.098152] LustreError: 20188:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 626.166162] LustreError: 19440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051155 with bad export cookie 7410699505707978932 [ 626.177482] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 626.183623] LustreError: 19440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 626.481658] Lustre: server umount lustre-MDT0001 complete [ 637.452571] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 640.668220] Key type lgssc unregistered [ 641.045775] LNet: 23012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.048851] LNetError: 23012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.063215] LNet: Removed LNI 192.168.201.157@tcp [ 642.016179] Key type .llcrypt unregistered [ 642.017602] Key type ._llcrypt unregistered [ 664.760302] Key type ._llcrypt registered [ 664.761622] Key type .llcrypt registered [ 664.831530] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 665.539726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 665.596931] alg: No test for adler32 (adler32-zlib) [ 666.651253] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 666.878460] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 668.535351] Key type lgssc registered [ 670.073707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 678.468651] loop: module loaded [ 678.526134] loop0: detected capacity change from 0 to 204800000 [ 678.701189] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.717533] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 679.243136] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.892993] loop0: detected capacity change from 0 to 204800000 [ 686.026393] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.037723] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.055398] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.603932] loop0: detected capacity change from 0 to 204800000 [ 696.749811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 696.812576] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.177443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 698.257150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 698.353856] Lustre: lustre-MDT0000: new disk, initializing [ 698.507948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.547846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 703.696721] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 713.745892] loop1: detected capacity change from 0 to 204800000 [ 719.514595] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 719.835993] Lustre: lustre-OST07c6: new disk, initializing [ 719.840856] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 719.845442] Lustre: Skipped 1 previous similar message [ 719.854524] Lustre: 26199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 719.945244] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 725.665942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 725.681457] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 725.720911] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 726.717461] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 732.302580] Lustre: 25185:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 732.309476] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 732.315845] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 732.321200] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 732.338074] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 732.347583] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 749.581144] Lustre: 25186:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 749.590183] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 749.599781] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 749.608138] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 749.617842] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 749.628881] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.426088] Lustre: 25186:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 754.435711] Lustre: 25186:0:(osd_internal.h:1452:osd_trans_exec_op()) Skipped 1 previous similar message [ 754.440315] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 754.446530] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.451379] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.455806] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.461530] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 754.468376] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.472471] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 754.477275] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.481649] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.485812] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.697732] Lustre: 25186:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 757.707475] Lustre: 25186:0:(osd_internal.h:1452:osd_trans_exec_op()) Skipped 1 previous similar message [ 757.711179] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 757.714252] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.718128] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 757.721208] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.728478] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 757.737983] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.747430] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 757.752337] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.759227] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 757.765664] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.456374] Lustre: 25185:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 763.468992] Lustre: 25185:0:(osd_internal.h:1452:osd_trans_exec_op()) Skipped 1 previous similar message [ 763.475205] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.479482] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.486209] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.492985] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.502027] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 763.510047] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.515766] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.525616] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.532557] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.541824] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.442734] Lustre: 25184:0:(osd_internal.h:1452:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 774.449807] Lustre: 25184:0:(osd_internal.h:1452:osd_trans_exec_op()) Skipped 5 previous similar messages [ 774.462474] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 774.477864] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.487608] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 774.493854] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.505364] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 774.513577] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.521553] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 774.528929] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.535813] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 774.539824] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 823.269303] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 823.275702] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.292907] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 827.875318] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 829.194942] Lustre: server umount lustre-OST07c6 complete [ 839.001077] Lustre: server umount lustre-MDT0000 complete [ 855.898440] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 859.011883] Key type lgssc unregistered [ 859.254437] LNet: 28755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.260572] LNetError: 28755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.328040] LNet: Removed LNI 192.168.201.157@tcp [ 861.210723] Key type .llcrypt unregistered [ 861.213525] Key type ._llcrypt unregistered [ 879.934600] Key type ._llcrypt registered [ 879.938325] Key type .llcrypt registered [ 880.054743] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 881.291868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.316191] alg: No test for adler32 (adler32-zlib) [ 882.512177] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 882.761932] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 884.455692] Key type lgssc registered [ 885.624641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 894.329451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 894.342285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.777213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.059905] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 905.189650] LustreError: 30023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.197662] LustreError: 30023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 908.042180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.327501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 913.132312] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 919.858678] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 925.617681] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 931.880978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 932.113694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 937.823564] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 940.352635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 945.556773] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 958.947026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.968260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.998703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.006110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 961.024688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.046021] Lustre: Skipped 1 previous similar message [ 964.981385] Lustre: server umount lustre-OST0000 complete [ 969.697023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 969.717399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.737284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 976.073889] Lustre: server umount lustre-MDT0000 complete [ 976.366761] LustreError: 30018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.426064] LustreError: 30018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 981.488802] LustreError: 30023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 981.524113] LustreError: 30023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 981.579143] LustreError: 33008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051510 with bad export cookie 16960672681353466921 [ 981.583799] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 981.594927] LustreError: 33008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 982.005261] Lustre: server umount lustre-MDT0001 complete [ 993.416242] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 996.599905] Key type lgssc unregistered [ 996.907996] LNet: 33759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.925430] LNetError: 33759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.950845] LNet: Removed LNI 192.168.201.157@tcp [ 997.916525] Key type .llcrypt unregistered [ 997.918411] Key type ._llcrypt unregistered [ 1018.603539] Key type ._llcrypt registered [ 1018.607860] Key type .llcrypt registered [ 1018.774115] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1019.893899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1020.035399] alg: No test for adler32 (adler32-zlib) [ 1021.247521] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 1021.608786] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1023.519260] Key type lgssc registered [ 1024.909989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1033.140220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1033.152604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1034.819235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.582825] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1043.941485] LustreError: 35028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1043.973671] LustreError: 35028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1048.180634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.482570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1052.536467] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1060.234581] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1064.982390] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1072.832661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1073.066225] LustreError: 36849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.451849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.794951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1080.701444] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1088.138143] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1099.235521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1099.260088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.274295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.256200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1100.259478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.264726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.292437] Lustre: Skipped 1 previous similar message [ 1105.395809] LustreError: 36852:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1105.527747] Lustre: server umount lustre-OST0000 complete [ 1110.507643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1110.516634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1110.517891] Lustre: Skipped 1 previous similar message [ 1110.535261] Lustre: Skipped 1 previous similar message [ 1115.616754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.629377] Lustre: Skipped 1 previous similar message [ 1116.849521] Lustre: server umount lustre-MDT0000 complete [ 1120.744036] LustreError: 35028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.791544] LustreError: 35028:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1122.793222] LustreError: 35007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051651 with bad export cookie 7371754250484449710 [ 1122.798445] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1122.809301] LustreError: 35007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1123.205426] Lustre: server umount lustre-MDT0001 complete [ 1139.912690] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1143.442266] Key type lgssc unregistered [ 1143.807766] LNet: 38562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.815522] LNetError: 38562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.852146] LNet: Removed LNI 192.168.201.157@tcp [ 1144.864979] Key type .llcrypt unregistered [ 1144.866301] Key type ._llcrypt unregistered [ 1172.112606] Key type ._llcrypt registered [ 1172.114183] Key type .llcrypt registered [ 1172.244687] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1173.667225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1173.674855] alg: No test for adler32 (adler32-zlib) [ 1174.833991] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 1175.094648] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1176.767450] Key type lgssc registered [ 1177.805836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1185.573975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1185.589902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1187.310972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1191.767687] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1197.540509] LustreError: 39976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1197.549532] LustreError: 39976:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1200.245364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.599443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1204.919195] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1212.576276] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1218.336815] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1225.512607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.830708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.996961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1232.435604] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1239.765184] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1246.281037] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 1246.690479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1246.696558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.344392] Lustre: server umount lustre-MDT0000 complete [ 1247.712675] LustreError: 39976:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.717274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.755490] Lustre: Skipped 1 previous similar message [ 1251.288645] LustreError: 39958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051780 with bad export cookie 9985193693167078979 [ 1251.317632] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1251.377213] LustreError: 41626:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.397983] LustreError: 41626:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1251.412800] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.57@tcp (stopping) [ 1251.417779] Lustre: Skipped 1 previous similar message [ 1252.832159] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.856066] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1256.494302] LustreError: 39971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.508702] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.57@tcp (stopping) [ 1256.515952] LustreError: 39971:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1257.706874] Lustre: server umount lustre-MDT0001 complete [ 1287.135138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1287.331600] Lustre: server umount lustre-OST0000 complete [ 1305.410969] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1308.382867] Key type lgssc unregistered [ 1308.797784] LNet: 43833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.801666] LNetError: 43833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.863182] LNet: Removed LNI 192.168.201.157@tcp [ 1310.739424] Key type .llcrypt unregistered [ 1310.742205] Key type ._llcrypt unregistered [ 1334.202640] Key type ._llcrypt registered [ 1334.206527] Key type .llcrypt registered [ 1334.383474] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1336.002180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1336.035846] alg: No test for adler32 (adler32-zlib) [ 1337.092617] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 1337.376847] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1339.136916] Key type lgssc registered [ 1340.387104] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1348.298558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1348.309405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.808145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.575901] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1359.336509] LustreError: 45248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.351543] LustreError: 45248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1363.276907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.915117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1369.373921] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1377.623384] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1383.437927] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1389.878865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1390.229387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.359118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1396.734920] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1404.740383] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1415.145373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1415.153441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.167537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1417.809801] Lustre: lustre-OST0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 1419.238994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.240571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1419.246919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1420.653240] Lustre: server umount lustre-OST0000 complete [ 1439.727189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1439.740969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.752622] Lustre: Skipped 2 previous similar messages [ 1445.273227] Lustre: server umount lustre-MDT0000 complete [ 1445.863611] LustreError: 45249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1445.877556] LustreError: 45249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1449.330308] LustreError: 45230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789051978 with bad export cookie 16835276598265472828 [ 1449.344919] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1455.960231] Lustre: server umount lustre-MDT0001 complete [ 1468.011541] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1471.518945] Key type lgssc unregistered [ 1471.803586] LNet: 48956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.813824] LNetError: 48956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.859844] LNet: Removed LNI 192.168.201.157@tcp [ 1473.144320] Key type .llcrypt unregistered [ 1473.148671] Key type ._llcrypt unregistered [ 1500.110829] Key type ._llcrypt registered [ 1500.113799] Key type .llcrypt registered [ 1500.291734] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1501.450321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1501.618736] alg: No test for adler32 (adler32-zlib) [ 1502.756982] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 1503.025231] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1504.679229] Key type lgssc registered [ 1505.721603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1513.453233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1513.469672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.153479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.725624] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1525.730786] LustreError: 50371:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.762419] LustreError: 50371:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1530.849818] LustreError: 50372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1532.580501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.921737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1537.881243] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1547.277421] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1554.224410] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1562.127687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1562.615788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.500439] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1577.130299] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1585.170993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1594.109691] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1597.915191] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1610.963990] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1010h55m38s [ 1612.630408] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1614.823504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1614.834695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.847989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.908139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.915595] Lustre: Skipped 5 previous similar messages [ 1621.031086] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 1621.050257] Lustre: Skipped 1 previous similar message [ 1627.189753] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 1627.200017] Lustre: Skipped 3 previous similar messages [ 1629.151521] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1629.354927] Lustre: server umount lustre-MDT0000 complete [ 1633.325206] LustreError: 50367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.576711] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1639.475198] LustreError: 52864:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.502741] LustreError: 52864:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1641.527493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.750455] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1642.123981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.213958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1643.177990] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1646.803675] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1650.231722] Lustre: DEBUG MARKER: Wait for df (36752) ... [ 1651.858561] Lustre: DEBUG MARKER: done [ 1657.317026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1657.322049] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.337503] Lustre: Skipped 2 previous similar messages [ 1657.344303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1657.353438] Lustre: Skipped 3 previous similar messages [ 1658.855489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.771100] Lustre: server umount lustre-OST0000 complete [ 1669.089340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.104822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1669.108364] Lustre: Skipped 3 previous similar messages [ 1681.375363] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1681.533486] Lustre: server umount lustre-MDT0000 complete [ 1685.267707] LustreError: 50353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052214 with bad export cookie 15274262692550628497 [ 1685.274350] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1685.285276] LustreError: 50353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1685.629853] Lustre: server umount lustre-MDT0001 complete [ 1698.858871] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1702.096333] Key type lgssc unregistered [ 1702.432639] LNet: 55322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.438727] LNetError: 55322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.482597] LNet: Removed LNI 192.168.201.157@tcp [ 1703.491935] Key type .llcrypt unregistered [ 1703.497754] Key type ._llcrypt unregistered [ 1745.695654] Key type ._llcrypt registered [ 1745.698368] Key type .llcrypt registered [ 1745.814390] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1748.533864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1748.660336] alg: No test for adler32 (adler32-zlib) [ 1750.041844] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 1750.487262] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1752.271275] Key type lgssc registered [ 1754.259221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1766.842924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1766.856964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.139868] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1768.159288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1768.291451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.520697] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1783.571642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.699049] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1783.739767] Lustre: 57529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1784.090934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1789.434801] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1797.086885] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1803.015532] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1811.243778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1811.434228] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1811.613592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.951583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1820.104785] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 1829.601166] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1837.727654] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1010h59m26s [ 1839.657345] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1848.616495] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1852.301203] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1852.369536] LustreError: 57536:0:(mdt_handler.c:2357:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1872.491036] Lustre: lustre-MDT0000: Client b8ae66b1-9caf-4c90-8299-eae7e7cfc6b4 (at 192.168.201.57@tcp) reconnecting [ 1892.455925] LustreError: 57536:0:(mdt_handler.c:2357:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1895.526820] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1898.497974] Lustre: DEBUG MARKER: done [ 1907.174281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1907.178856] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.181658] LustreError: Skipped 1 previous similar message [ 1907.201097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1910.257526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1910.266159] Lustre: Skipped 2 previous similar messages [ 1911.943379] Lustre: server umount lustre-OST0000 complete [ 1917.414549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1917.420642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.442196] Lustre: Skipped 1 previous similar message [ 1917.457661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.651892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.656389] Lustre: Skipped 4 previous similar messages [ 1922.478576] Lustre: server umount lustre-MDT0000 complete [ 1926.509823] LustreError: 60291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052455 with bad export cookie 8185174931460905586 [ 1926.513271] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1926.520733] LustreError: 60291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1926.967251] Lustre: server umount lustre-MDT0001 complete [ 1941.161775] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 1944.547198] Key type lgssc unregistered [ 1944.999117] LNet: 61874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.005197] LNetError: 61874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.023705] LNet: Removed LNI 192.168.201.157@tcp [ 1945.865439] Key type .llcrypt unregistered [ 1945.867304] Key type ._llcrypt unregistered [ 1994.285257] Key type ._llcrypt registered [ 1994.287708] Key type .llcrypt registered [ 1994.400353] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 1995.745307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1995.825940] alg: No test for adler32 (adler32-zlib) [ 1996.901251] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 1997.068912] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 1998.703287] Key type lgssc registered [ 1999.911713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2003.375546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2003.410731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.794624] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2004.813411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2004.971400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.887762] Lustre: Failing over lustre-MDT0000 [ 2006.328931] Lustre: server umount lustre-MDT0000 complete [ 2013.966424] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:02:22 (1789052542) [ 2020.915123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.390267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.323631] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2034.685340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.749340] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2034.771980] Lustre: 64679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2035.049920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2040.409441] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2051.385986] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2059.180359] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2067.933291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2068.499467] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2068.934531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.891371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2077.445345] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2085.990334] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2092.456792] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2099.331464] Lustre: server umount lustre-MDT0000 complete [ 2101.743039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2101.758627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.753494] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.665725] LustreError: 64671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052632 with bad export cookie 16418057626951039706 [ 2103.669152] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2103.681888] LustreError: 64671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2104.285164] Lustre: server umount lustre-MDT0001 complete [ 2108.347936] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2113.692764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2123.178812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2124.255276] Lustre: 63603:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789052637/real 1789052637] req@000000004cb3a47b x1875957541546880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789052653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2124.292826] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.309878] Lustre: Skipped 1 previous similar message [ 2125.280304] Lustre: 63604:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789052638/real 1789052638] req@000000004b690247 x1875957541547008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789052654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.415642] LustreError: 63600:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f17383bd x1875957541548160/t0(0) o250->MGC192.168.201.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.832615] LustreError: 66529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.857984] LustreError: 66529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2128.918402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2129.079613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2129.887144] Lustre: 63601:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789052642/real 1789052642] req@000000007280ed1e x1875957541547136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789052658 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2129.980857] LustreError: 66530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.001495] Lustre: 63602:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789052645/real 1789052645] req@0000000029bb6bc2 x1875957541547392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789052661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2134.050693] LustreError: 68448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.075670] LustreError: 68448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2134.317958] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2138.143134] Lustre: 63603:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789052651/real 1789052651] req@000000007ed67471 x1875957541547648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789052667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2138.183170] Lustre: 63603:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2139.127991] LustreError: 68447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.159037] LustreError: 68447:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2142.438429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.758291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2143.787766] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2147.204351] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2155.992690] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2163.707869] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2173.410306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2173.436036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.453522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.437056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.724609] Lustre: server umount lustre-MDT0000 complete [ 2179.559151] LustreError: 69508:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.605704] LustreError: 69508:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2183.766730] LustreError: 64671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052712 with bad export cookie 16418057626951041981 [ 2183.804904] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2184.169954] Lustre: server umount lustre-MDT0001 complete [ 2192.777616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2200.038074] Lustre: 63602:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789052713/real 1789052713] req@00000000fcfcf8ed x1875957541576832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789052729 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2200.069343] Lustre: 63602:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2200.073471] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.085201] Lustre: Skipped 2 previous similar messages [ 2202.329633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.313453] LustreError: 63600:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ef064ed5 x1875957541577856/t0(0) o250->MGC192.168.201.157@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2209.863606] LustreError: 66529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2209.940480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.019803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2215.854909] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2226.657893] LustreError: 66530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.687988] LustreError: 66530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 2226.911358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2229.298532] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2229.318341] Lustre: Skipped 1 previous similar message [ 2233.549902] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2242.464265] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2248.366553] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2257.889198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2257.898464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2257.924146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2257.931214] Lustre: Skipped 2 previous similar messages [ 2259.948685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2259.959443] Lustre: Skipped 2 previous similar messages [ 2262.696158] Lustre: server umount lustre-MDT0000 complete [ 2265.061449] LustreError: 66530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2265.089354] LustreError: 66530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2267.279077] LustreError: 66535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052796 with bad export cookie 16418057626951043360 [ 2267.284834] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2267.295163] LustreError: 66535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2267.798468] Lustre: server umount lustre-MDT0001 complete [ 2275.056782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2278.713266] Lustre: DEBUG MARKER: files compared the same [ 2290.186497] Lustre: server umount lustre-OST0000 complete [ 2308.172653] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 2311.326418] Key type lgssc unregistered [ 2311.654740] LNet: 74409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.664454] LNetError: 74409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.678557] LNet: Removed LNI 192.168.201.157@tcp [ 2313.822616] Key type .llcrypt unregistered [ 2313.825994] Key type ._llcrypt unregistered [ 2338.743451] Key type ._llcrypt registered [ 2338.746341] Key type .llcrypt registered [ 2338.869967] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 2339.977691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2340.021670] alg: No test for adler32 (adler32-zlib) [ 2341.247293] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 2341.526372] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 2343.223247] Key type lgssc registered [ 2344.503334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2352.321386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2352.368820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.001648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.565851] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug +malloc all [ 2364.386225] LustreError: 75820:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.402025] LustreError: 75820:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2366.948043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.344716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2371.680456] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug +malloc all [ 2379.292752] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2385.379680] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2393.416052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2393.811784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.954684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2400.436874] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug +malloc all [ 2408.983693] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2422.756103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2422.763803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.781978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2422.793454] Lustre: Skipped 1 previous similar message [ 2426.037722] Lustre: server umount lustre-OST0000 complete [ 2432.993336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2433.011750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2433.024590] Lustre: Skipped 2 previous similar messages [ 2435.804162] Lustre: server umount lustre-MDT0000 complete [ 2438.115311] LustreError: 75821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.139899] LustreError: 75821:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2439.841479] LustreError: 75800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789052969 with bad export cookie 6375660640483632784 [ 2439.860430] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2440.304817] Lustre: server umount lustre-MDT0001 complete [ 2453.038108] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 2456.730436] Key type lgssc unregistered [ 2457.178788] LNet: 79352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.193176] LNetError: 79352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.216295] LNet: Removed LNI 192.168.201.157@tcp [ 2458.530848] Key type .llcrypt unregistered [ 2458.537812] Key type ._llcrypt unregistered [ 2489.987680] Key type ._llcrypt registered [ 2489.989691] Key type .llcrypt registered [ 2490.074472] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 2491.590869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2491.612507] alg: No test for adler32 (adler32-zlib) [ 2492.686287] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 2492.978541] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 2494.663273] Key type lgssc registered [ 2495.953781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2504.023584] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2504.041853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2519.583533] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2524.703363] LustreError: 80741:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.157@tcp: failed processing log, type 4: rc = -110 [ 2551.455231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.355955] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2563.640776] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2571.860167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.031367] Lustre: *** cfs_fail_loc=706, val=0*** [ 2572.741363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.859604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2576.436552] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2577.891692] LustreError: 80766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.914438] LustreError: 80766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2583.013170] LustreError: 80767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.701361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2586.082969] LustreError: 80766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.122704] LustreError: 80766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2586.586522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2591.695469] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2600.191704] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2607.460834] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2612.192307] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2612.205771] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.214921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2612.219995] Lustre: Skipped 1 previous similar message [ 2612.238199] Lustre: Skipped 1 previous similar message [ 2615.522337] Lustre: server umount lustre-OST0000 complete [ 2622.435152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2622.442752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2622.448589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2624.900891] Lustre: server umount lustre-MDT0000 complete [ 2627.557813] LustreError: 81830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.427157] LustreError: 80748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053157 with bad export cookie 16864565271083643041 [ 2628.444195] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2628.826227] Lustre: server umount lustre-MDT0001 complete [ 2639.143686] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 2643.043664] Key type lgssc unregistered [ 2643.365624] LNet: 84357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2643.380821] LNetError: 84357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2644.457282] LNet: Removed LNI 192.168.201.157@tcp [ 2645.341724] Key type .llcrypt unregistered [ 2645.343841] Key type ._llcrypt unregistered [ 2666.364772] Key type ._llcrypt registered [ 2666.367524] Key type .llcrypt registered [ 2666.631553] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 2668.485282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2668.522071] alg: No test for adler32 (adler32-zlib) [ 2669.939949] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 2670.321348] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 2672.095270] Key type lgssc registered [ 2673.518331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2680.404403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2680.418967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.411237] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2694.533970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.215537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2700.210739] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2700.261809] LustreError: 86438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.282721] LustreError: 86438:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2705.386366] LustreError: 86442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.396160] LustreError: 86442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2709.117605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2709.566624] LustreError: 86438:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.587372] LustreError: 86438:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2709.599930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.594640] LustreError: 86442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.624506] LustreError: 86442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2716.305772] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2719.730395] LustreError: 86441:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.767052] LustreError: 86441:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2723.013787] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2723.034507] Lustre: MGS: Client b44ebec4-53f5-449e-bf4c-48fa0051c537 (at 0@lo) reconnecting [ 2723.063984] LustreError: 86452:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2723.293411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.344778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2727.218037] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2743.780747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2743.794615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.839204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2748.649291] Lustre: server umount lustre-OST0000 complete [ 2754.019273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2754.020329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.047291] Lustre: Skipped 1 previous similar message [ 2754.059343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.064954] Lustre: Skipped 1 previous similar message [ 2758.746184] Lustre: server umount lustre-MDT0000 complete [ 2759.136815] LustreError: 86442:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.164134] LustreError: 86442:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2762.406392] Lustre: server umount lustre-MDT0001 complete [ 2765.724069] Lustre: server umount MGS complete [ 2778.044813] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:15:06 (1789053306) [ 2803.757585] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 2812.038295] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 2849.819036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.586867] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2861.529914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.668772] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2861.698598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2861.751586] Lustre: lustre-MDT0001: new disk, initializing [ 2861.800414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2861.843295] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2861.857702] Lustre: 93623:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2861.866543] LustreError: 93623:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2866.992418] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2876.713039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2877.050858] Lustre: lustre-OST0000: new disk, initializing [ 2877.056383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2877.069183] Lustre: 94390:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2877.178292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2883.477766] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2883.692373] Lustre: 94685:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004b8fd52c x1875958247235200/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789053428 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2891.188203] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2897.167264] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.174702] Lustre: MGS: Client c9c4cccd-09bd-4b48-9f84-668532b2be8e (at 0@lo) reconnecting [ 2897.198160] LustreError: 93609:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2897.246972] Lustre: 92893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2897.313955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2897.418873] Lustre: lustre-MDT0000: new disk, initializing [ 2897.565176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.594630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2897.707907] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2898.655975] Lustre: 95458:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006876dd4e x1875958247241728/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789053443 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2902.018766] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 2904.036118] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2907.749052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2907.753862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2907.898986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2933.742948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2933.750304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.759597] Lustre: Skipped 1 previous similar message [ 2933.763569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2938.006621] Lustre: server umount lustre-OST0000 complete [ 2945.505901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2945.507045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2945.530996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2945.550733] Lustre: Skipped 2 previous similar messages [ 2945.569150] Lustre: Skipped 3 previous similar messages [ 2947.859281] Lustre: server umount lustre-MDT0000 complete [ 2950.625555] LustreError: 93598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.646232] LustreError: 93598:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2958.121520] Lustre: server umount lustre-MDT0001 complete [ 2962.236794] Lustre: server umount MGS complete [ 2978.620469] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:18:27 (1789053507) [ 2998.194525] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3001.394531] Key type lgssc unregistered [ 3001.729840] LNet: 97960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3001.737948] LNetError: 97960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3001.761606] LNet: Removed LNI 192.168.201.157@tcp [ 3002.712850] Key type .llcrypt unregistered [ 3002.715445] Key type ._llcrypt unregistered [ 3020.347025] Key type ._llcrypt registered [ 3020.348452] Key type .llcrypt registered [ 3020.513522] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3021.810024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3021.832532] alg: No test for adler32 (adler32-zlib) [ 3023.024297] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 3023.256461] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3025.024575] Key type lgssc registered [ 3026.186179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3036.132909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3036.177340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.869726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.391219] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3047.396768] LustreError: 99180:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3047.414986] LustreError: 99180:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3051.068347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.275693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3055.290421] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3062.703668] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3068.587708] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3074.916112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3075.254312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3081.116488] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3082.415360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3088.066793] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3096.523926] Lustre: server umount lustre-OST0000 complete [ 3102.688381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3102.695576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3102.703628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.204487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.216870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.222547] Lustre: Skipped 1 previous similar message [ 3108.320598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.327400] Lustre: Skipped 1 previous similar message [ 3110.471881] Lustre: server umount lustre-MDT0000 complete [ 3113.439876] Lustre: 98489:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789053626/real 1789053626] req@000000008ebc0f96 x1875958617367552/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789053642 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3113.446473] LustreError: 100229:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.481969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3113.526723] LustreError: 100229:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3114.368049] LustreError: 99161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053643 with bad export cookie 16695125315553533130 [ 3114.369126] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3114.377352] LustreError: 99161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3114.703221] Lustre: server umount lustre-MDT0001 complete [ 3126.499650] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3129.472658] Key type lgssc unregistered [ 3129.779831] LNet: 102714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3129.785939] LNetError: 102714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3130.866492] LNet: Removed LNI 192.168.201.157@tcp [ 3131.701818] Key type .llcrypt unregistered [ 3131.707120] Key type ._llcrypt unregistered [ 3147.928471] Key type ._llcrypt registered [ 3147.929937] Key type .llcrypt registered [ 3148.052975] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3149.135782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3149.300099] alg: No test for adler32 (adler32-zlib) [ 3150.548524] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 3150.762308] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3152.488411] Key type lgssc registered [ 3153.587327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3159.794697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3159.812544] LustreError: 103803:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3159.814943] LustreError: 103804:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3159.838671] LustreError: 103803:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3159.843884] LustreError: 103803:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3159.856467] LustreError: 103803:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3159.866363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3161.304447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.416898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3168.636152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3172.743874] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3179.985985] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing lsmod [ 3183.718732] LustreError: 105008:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3183.753447] LustreError: 105009:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3183.761327] LustreError: 105008:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4975 [ 3183.770643] LustreError: 105008:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3183.783728] LustreError: 105008:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3183.797533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3184.473873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3186.578428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3189.220023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3189.233306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.255539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.776677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3191.778174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3191.795061] Lustre: Skipped 1 previous similar message [ 3191.807237] Lustre: Skipped 1 previous similar message [ 3192.876844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3192.889482] Lustre: Skipped 3 previous similar messages [ 3194.261719] Lustre: server umount lustre-MDT0000 complete [ 3197.927079] LustreError: 103836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.950823] LustreError: 103836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3198.264337] LustreError: 103816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053727 with bad export cookie 413240952926874602 [ 3198.267589] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3198.276903] LustreError: 103816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3199.074961] LustreError: 104272:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000677e3225 x1875958750534144/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3199.282878] Lustre: server umount lustre-MDT0001 complete [ 3213.337266] Lustre: server umount lustre-OST0000 complete [ 3222.859176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.343190] LustreError: 106072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.363992] LustreError: 106072:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3223.439331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.536507] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3228.643312] LustreError: 106073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3233.774347] LustreError: 106072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.811832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.235665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3240.948193] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3248.920591] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3256.144408] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3263.974952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.435883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3268.539765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3271.281295] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3278.697199] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3292.127988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3292.134170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3292.156330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3292.170587] Lustre: Skipped 1 previous similar message [ 3296.178888] Lustre: server umount lustre-OST0000 complete [ 3302.372673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.376448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.395367] Lustre: Skipped 1 previous similar message [ 3302.420019] Lustre: Skipped 2 previous similar messages [ 3308.121384] Lustre: server umount lustre-MDT0000 complete [ 3312.608528] LustreError: 106073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.639509] LustreError: 106073:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3312.669121] LustreError: 106052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789053841 with bad export cookie 413240952926875533 [ 3312.697634] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3312.707731] LustreError: 106052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3313.339748] Lustre: server umount lustre-MDT0001 complete [ 3326.833171] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3331.007894] Key type lgssc unregistered [ 3331.432960] LNet: 109605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3331.442294] LNetError: 109605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3332.463504] LNet: Removed LNI 192.168.201.157@tcp [ 3333.980888] Key type .llcrypt unregistered [ 3333.983905] Key type ._llcrypt unregistered [ 3360.259479] Key type ._llcrypt registered [ 3360.261046] Key type .llcrypt registered [ 3360.424254] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3361.781357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3361.868187] alg: No test for adler32 (adler32-zlib) [ 3362.866735] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 3363.076222] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3364.711805] Key type lgssc registered [ 3365.909948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3374.151528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3374.177731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.802101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.595842] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3385.319120] LustreError: 111019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.370835] LustreError: 111019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3390.435847] LustreError: 111020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.034440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.618325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3396.608855] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3403.788475] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3409.870328] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3418.267968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.816721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3419.899864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3427.958851] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3436.242596] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3441.546215] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3444.649647] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3450.920827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3450.930515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3450.962658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3452.900354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3452.904053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.511735] Lustre: server umount lustre-OST0000 complete [ 3459.044846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3459.054174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.071973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3459.079612] Lustre: Skipped 1 previous similar message [ 3463.153331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.154524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3463.182667] Lustre: Skipped 1 previous similar message [ 3468.264952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3468.268542] Lustre: Skipped 1 previous similar message [ 3473.381232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3473.566129] Lustre: server umount lustre-MDT0000 complete [ 3478.212342] LustreError: 110999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054007 with bad export cookie 9466890550045162718 [ 3478.216509] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3478.222558] LustreError: 110999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3478.539527] Lustre: server umount lustre-MDT0001 complete [ 3494.040602] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3497.407451] Key type lgssc unregistered [ 3497.785894] LNet: 114657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3497.795737] LNetError: 114657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3497.812345] LNet: Removed LNI 192.168.201.157@tcp [ 3498.965750] Key type .llcrypt unregistered [ 3498.968762] Key type ._llcrypt unregistered [ 3516.022488] Key type ._llcrypt registered [ 3516.026485] Key type .llcrypt registered [ 3516.192985] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3517.636554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3517.689325] alg: No test for adler32 (adler32-zlib) [ 3518.815190] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 3519.144256] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3520.887590] Key type lgssc registered [ 3522.009632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3529.397643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3529.415864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.886832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.043452] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3541.473804] LustreError: 115878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.493170] LustreError: 115878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3544.577483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.883322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3549.386858] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3555.633979] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3560.700421] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3566.577317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.883078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.354229] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3576.116735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3579.415074] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3589.614930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3589.629243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.668183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3590.137169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3590.143356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.650281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3591.656905] Lustre: Skipped 2 previous similar messages [ 3596.771676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.790510] Lustre: Skipped 1 previous similar message [ 3601.890852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3601.909422] Lustre: Skipped 1 previous similar message [ 3602.399183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3602.763447] Lustre: server umount lustre-OST0000 complete [ 3607.010681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3607.029237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3607.043198] Lustre: Skipped 1 previous similar message [ 3612.951810] Lustre: server umount lustre-MDT0000 complete [ 3616.602901] LustreError: 115858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054145 with bad export cookie 13748767380142647552 [ 3616.609289] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3616.614285] LustreError: 115858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3616.986319] Lustre: server umount lustre-MDT0001 complete [ 3627.875839] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3630.717714] Key type lgssc unregistered [ 3631.028902] LNet: 119409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3631.044469] LNetError: 119409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3631.061182] LNet: Removed LNI 192.168.201.157@tcp [ 3631.851089] Key type .llcrypt unregistered [ 3631.852287] Key type ._llcrypt unregistered [ 3654.375414] Key type ._llcrypt registered [ 3654.378438] Key type .llcrypt registered [ 3654.485577] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3655.459844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3655.627100] alg: No test for adler32 (adler32-zlib) [ 3656.635392] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 3656.802591] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3658.439318] Key type lgssc registered [ 3659.460948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3666.668600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3666.691731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3668.130371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3671.860796] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3678.698150] LustreError: 120822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3678.710464] LustreError: 120822:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3683.015470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3683.813254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3689.853410] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3697.157195] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3702.711649] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3709.409349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3709.741858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.854155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3715.433973] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3723.367654] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3732.423658] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3736.683788] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3736.689335] Lustre: Skipped 2 previous similar messages [ 3737.724792] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3737.760503] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3740.874932] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3740.880210] Lustre: Skipped 2 previous similar messages [ 3746.387985] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3746.394104] Lustre: Skipped 1 previous similar message [ 3751.648653] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.57@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3751.659915] Lustre: Skipped 2 previous similar messages [ 3752.372559] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.57@tcp 192.168.0.[2,10]@tcp [ 3755.889508] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.57/24@tcp in log lustre-MDT0000 [ 3755.902249] Lustre: Skipped 2 previous similar messages [ 3759.448842] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.57/24@tcp [ 3759.452752] Lustre: Skipped 1 previous similar message [ 3765.220259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3765.229696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.252536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3769.312546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.317605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3769.339382] Lustre: Skipped 1 previous similar message [ 3770.880641] Lustre: server umount lustre-OST0000 complete [ 3775.972383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3775.981889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3775.998732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3779.556595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3779.562177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.674382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3784.681095] Lustre: Skipped 2 previous similar messages [ 3790.305332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3790.742981] Lustre: server umount lustre-MDT0000 complete [ 3794.919181] LustreError: 123314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.948355] LustreError: 123314:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3797.910255] LustreError: 123453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054327 with bad export cookie 18272743335609749749 [ 3797.919794] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3797.923931] LustreError: 123453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3798.497561] Lustre: server umount lustre-MDT0001 complete [ 3810.973303] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3813.732142] Key type lgssc unregistered [ 3814.044891] LNet: 125623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3814.058646] LNetError: 125623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3815.077713] LNet: Removed LNI 192.168.201.157@tcp [ 3816.069339] Key type .llcrypt unregistered [ 3816.073366] Key type ._llcrypt unregistered [ 3848.120190] Key type ._llcrypt registered [ 3848.122667] Key type .llcrypt registered [ 3848.244507] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 3849.748122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3849.826629] alg: No test for adler32 (adler32-zlib) [ 3851.292867] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 3851.581678] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 3853.255777] Key type lgssc registered [ 3855.262521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3864.778809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3864.819444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.354470] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3866.360024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3866.375443] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.57/24@tcp [ 3869.955281] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3876.836051] LustreError: 127180:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3876.846205] LustreError: 127180:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3877.585837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.786366] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3877.816730] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.57/24@tcp [ 3877.850868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3882.327361] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3889.640980] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3895.233084] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3904.080208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3904.461157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3909.807865] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 3912.706425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3920.640447] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3937.265572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3937.272854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.284665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3937.301202] Lustre: Skipped 1 previous similar message [ 3939.962050] Lustre: server umount lustre-OST0000 complete [ 3944.418867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3944.434675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.473050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3944.483617] Lustre: Skipped 1 previous similar message [ 3947.488612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3947.492740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3952.610173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3952.621262] Lustre: Skipped 2 previous similar messages [ 3957.729731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3957.744320] Lustre: Skipped 1 previous similar message [ 3958.239164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3958.498605] Lustre: server umount lustre-MDT0000 complete [ 3962.853699] LustreError: 128213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3962.872417] LustreError: 128213:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3963.455930] LustreError: 127160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054492 with bad export cookie 661298994777638129 [ 3963.469137] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.477769] LustreError: 127160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3963.875631] Lustre: server umount lustre-MDT0001 complete [ 3976.324843] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 3980.452738] Key type lgssc unregistered [ 3981.033784] LNet: 130772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3981.046572] LNetError: 130772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3981.073092] LNet: Removed LNI 192.168.201.157@tcp [ 3982.262114] Key type .llcrypt unregistered [ 3982.265991] Key type ._llcrypt unregistered [ 4011.812791] Key type ._llcrypt registered [ 4011.815506] Key type .llcrypt registered [ 4011.952585] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 4028.581686] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4029.777604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4029.831927] alg: No test for adler32 (adler32-zlib) [ 4030.884489] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 4031.254422] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4032.983364] Key type lgssc registered [ 4034.312834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4068.972676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4069.005162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.299471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4070.329911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4070.436974] Lustre: lustre-MDT0000: new disk, initializing [ 4070.546310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.573626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4075.053952] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4086.665925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.918226] Lustre: 134171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4087.026603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4087.034161] Lustre: Skipped 1 previous similar message [ 4087.154600] Lustre: lustre-MDT0001: new disk, initializing [ 4087.268131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4087.318655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4087.330856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4092.772881] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4102.432052] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4108.399208] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4116.070902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4116.562045] Lustre: lustre-OST0000: new disk, initializing [ 4116.578221] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4116.583846] Lustre: 136141:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4116.709604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4124.319369] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4126.294497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4126.306508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4126.406745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4134.189481] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4140.101914] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4140.515856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4145.706796] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4145.898757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4152.301367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.320442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4152.329918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4152.352491] Lustre: Skipped 1 previous similar message [ 4154.750166] Lustre: server umount lustre-OST0000 complete [ 4162.529580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.557175] Lustre: Skipped 2 previous similar messages [ 4162.557554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4162.585740] Lustre: Skipped 1 previous similar message [ 4165.627765] Lustre: server umount lustre-MDT0000 complete [ 4170.749467] LustreError: 134163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054699 with bad export cookie 18127809012604846063 [ 4170.750759] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4170.758111] LustreError: 134163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4171.134451] Lustre: server umount lustre-MDT0001 complete [ 4181.345880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4181.687802] LustreError: 138358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.705893] LustreError: 138358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4181.741412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.728335] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4187.110028] LustreError: 138359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4191.212257] LustreError: 138358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.057518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.325680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4196.558418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4202.102669] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4210.702407] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4216.987581] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4223.292650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4223.821395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.054964] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4240.453332] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4245.581854] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4245.907414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4251.614440] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4251.976097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4263.419892] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4263.622364] Lustre: lustre-OST0001: new disk, initializing [ 4263.626771] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4263.638123] Lustre: 141735:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4263.758787] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4270.203403] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4272.287206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4272.301399] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4272.421494] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4279.878049] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4280.457910] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4286.788227] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4287.002238] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4293.605669] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff989058ce7000.ost_server_uuid 50 [ 4294.950178] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff989058ce7000.ost_server_uuid in IDLE state after 0 sec [ 4308.458528] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4308.460369] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.469612] LustreError: Skipped 1 previous similar message [ 4308.476065] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4308.493451] Lustre: Skipped 1 previous similar message [ 4308.516425] Lustre: Skipped 1 previous similar message [ 4310.253945] Lustre: server umount lustre-OST0001 complete [ 4313.579950] LustreError: 140182:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4313.599454] LustreError: 140182:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4318.177264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4318.178183] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.194297] LustreError: Skipped 1 previous similar message [ 4318.195468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4318.213242] Lustre: Skipped 1 previous similar message [ 4318.240020] Lustre: Skipped 1 previous similar message [ 4318.699602] LustreError: 140873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4318.738494] LustreError: 140873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4320.324764] Lustre: server umount lustre-OST0000 complete [ 4324.832505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4324.841311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.928057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.939824] Lustre: Skipped 2 previous similar messages [ 4330.660491] Lustre: server umount lustre-MDT0000 complete [ 4334.049133] LustreError: 138354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.070676] LustreError: 138354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4334.579464] LustreError: 138339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789054863 with bad export cookie 18127809012604846945 [ 4334.584023] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4334.587990] LustreError: 138339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4335.095249] Lustre: server umount lustre-MDT0001 complete [ 4359.306432] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4362.309591] Key type lgssc unregistered [ 4362.796345] LNet: 144740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4362.818728] LNetError: 144740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4362.872448] LNet: Removed LNI 192.168.201.157@tcp [ 4364.478744] Key type .llcrypt unregistered [ 4364.481600] Key type ._llcrypt unregistered [ 4410.452366] Key type ._llcrypt registered [ 4410.453743] Key type .llcrypt registered [ 4410.558851] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 4427.817372] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4429.577372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4429.670126] alg: No test for adler32 (adler32-zlib) [ 4430.765868] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 4431.028949] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4432.711434] Key type lgssc registered [ 4434.217392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4471.940915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4471.969441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4473.432789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4473.462605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4473.532810] Lustre: lustre-MDT0000: new disk, initializing [ 4473.617341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.653483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4481.165525] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4494.331144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.508965] Lustre: 148894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4494.567535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4494.570862] Lustre: Skipped 1 previous similar message [ 4494.616307] Lustre: lustre-MDT0001: new disk, initializing [ 4494.690425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4494.722073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4494.731261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4500.153668] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4510.298677] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4517.072923] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4526.625634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4527.235290] Lustre: lustre-OST0000: new disk, initializing [ 4527.238805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4527.259164] Lustre: 150862:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4527.468153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4534.916632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4534.936385] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4535.071271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4536.221784] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4546.709934] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4550.327715] Lustre: Failing over lustre-OST0000 [ 4550.418170] Lustre: server umount lustre-OST0000 complete [ 4567.008409] Lustre: 147117:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055079/real 1789055079] req@00000000ae3e0028 x1875960093773440/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789055095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4567.043497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4569.121362] Lustre: 147118:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055082/real 1789055082] req@00000000b8db7e01 x1875960093773952/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789055098 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4569.148871] Lustre: 147118:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4571.211570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4571.231351] Lustre: 147118:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055084/real 1789055084] req@0000000071e7b0e6 x1875960093774848/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789055100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4571.609823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4571.644164] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4577.761825] Lustre: 147118:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055090/real 1789055090] req@0000000025de33a4 x1875960093775872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789055106 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4577.795975] Lustre: 147118:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4577.800188] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4579.493787] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4580.472848] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4580.500705] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4582.559103] Lustre: 147117:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055095/real 1789055095] req@00000000826cc07f x1875960093777024/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789055111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4582.587099] Lustre: 147117:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4583.926925] Lustre: Failing over lustre-MDT0000 [ 4584.120691] Lustre: server umount lustre-MDT0000 complete [ 4586.977052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4586.994713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4587.001477] Lustre: Skipped 1 previous similar message [ 4588.013142] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.025036] Lustre: Skipped 1 previous similar message [ 4590.630216] LustreError: 148901:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.636120] LustreError: 148901:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4593.127443] LustreError: 150870:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4593.149095] LustreError: 150870:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4595.759172] LustreError: 148901:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4598.252981] LustreError: 148905:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4598.316804] LustreError: 148905:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4603.368491] LustreError: 148904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4603.412183] LustreError: 148904:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4604.386515] Lustre: 147116:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789055117/real 1789055117] req@0000000071e7b0e6 x1875960093788672/t0(0) o400->MGC192.168.201.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789055133 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4604.456321] Lustre: 147116:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4604.467984] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4605.002561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.624591] LustreError: 148901:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4614.638321] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3c8fa6f8e4b0cd43 [ 4614.653892] LustreError: 148901:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 4614.676822] Lustre: MGC192.168.201.157@tcp: Connection restored to 0@lo (at 0@lo) [ 4614.686255] Lustre: Skipped 1 previous similar message [ 4614.972175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4615.017668] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4616.258536] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4620.291882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4620.316389] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4620.368861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4620.744173] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4632.033730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4632.042710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.061361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4635.623279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4637.680894] Lustre: server umount lustre-OST0000 complete [ 4645.857594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4645.883814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4645.905631] Lustre: Skipped 1 previous similar message [ 4645.929139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4645.934536] Lustre: Skipped 2 previous similar messages [ 4648.618408] Lustre: server umount lustre-MDT0000 complete [ 4650.975752] LustreError: 148904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4650.983034] LustreError: 148904:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4652.994853] LustreError: 150861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055182 with bad export cookie 4363890151864192323 [ 4653.013056] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4653.020070] LustreError: 150861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4653.959343] Lustre: server umount lustre-MDT0001 complete [ 4665.525732] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 4668.838415] Key type lgssc unregistered [ 4669.309533] LNet: 154522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4669.323531] LNetError: 154522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4669.359466] LNet: Removed LNI 192.168.201.157@tcp [ 4670.351308] Key type .llcrypt unregistered [ 4670.354547] Key type ._llcrypt unregistered [ 4709.543041] Key type ._llcrypt registered [ 4709.544909] Key type .llcrypt registered [ 4709.676759] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4710.851897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4710.910343] alg: No test for adler32 (adler32-zlib) [ 4712.035304] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 4712.360471] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 4714.031463] Key type lgssc registered [ 4715.936468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4727.421831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4727.453982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4729.438253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4735.454641] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4739.557732] LustreError: 156039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4739.586044] LustreError: 156039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4744.678089] LustreError: 156040:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4746.084766] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4749.793063] LustreError: 156039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4750.598277] LustreError: 156068:0:(osp_object.c:623:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4750.616340] LustreError: 156068:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4750.626414] LustreError: 156068:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4751.411552] Lustre: server umount lustre-MDT0000 complete [ 4768.897423] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 4783.095121] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 4821.663448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4821.957512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4822.000431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4822.149501] Lustre: lustre-MDT0000: new disk, initializing [ 4822.289900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4822.309958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4827.398916] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4838.734291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4838.964643] Lustre: 159392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4839.037864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4839.041404] Lustre: Skipped 1 previous similar message [ 4839.146165] Lustre: lustre-MDT0001: new disk, initializing [ 4839.217036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4839.240755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4839.245156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4845.771964] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4856.343642] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4862.712653] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4870.081286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4870.420257] Lustre: lustre-OST0000: new disk, initializing [ 4870.428284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4870.437369] Lustre: 161362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4870.536909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4876.340960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4876.351162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4876.477794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4877.131605] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 4887.478830] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4891.945245] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4894.463311] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:50:22 (1789055422) [ 5126.111869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5126.122377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5126.132523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5127.650314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5127.653176] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5130.382883] Lustre: server umount lustre-MDT0000 complete [ 5132.771938] LustreError: 159399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.807401] LustreError: 159399:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5136.037690] LustreError: 159383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055665 with bad export cookie 7525882113148203762 [ 5136.038510] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.056128] LustreError: 159383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5137.890337] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5137.890652] LustreError: 160678:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5137.900617] Lustre: Skipped 1 previous similar message [ 5137.924284] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5137.934452] Lustre: Skipped 2 previous similar messages [ 5141.919960] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5142.390474] Lustre: server umount lustre-MDT0001 complete [ 5151.743387] Lustre: server umount lustre-OST0000 complete [ 5162.087417] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 5171.575473] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 5212.784851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5213.112429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5213.182150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5213.319871] Lustre: lustre-MDT0000: new disk, initializing [ 5213.416096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5213.437132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5218.432301] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5229.701044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.845542] Lustre: 166024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5229.915842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5229.921683] Lustre: Skipped 1 previous similar message [ 5229.993140] Lustre: lustre-MDT0001: new disk, initializing [ 5230.064899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5230.101079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5230.118544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5235.114987] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5245.236314] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5252.011397] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5258.600362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5258.875679] Lustre: lustre-OST0000: new disk, initializing [ 5258.879131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5258.886694] Lustre: 167992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5258.956143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5266.657670] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5267.031163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5267.055511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5267.188611] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5278.535146] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5284.565427] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5284.780964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5289.998265] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5290.277376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5293.025780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5293.026406] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.042249] LustreError: Skipped 1 previous similar message [ 5293.045866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5293.111898] Lustre: Skipped 1 previous similar message [ 5298.656637] Lustre: server umount lustre-OST0000 complete [ 5303.265399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.307917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5303.325518] Lustre: Skipped 4 previous similar messages [ 5317.087131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5317.265131] Lustre: server umount lustre-MDT0000 complete [ 5318.625328] LustreError: 166824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5318.659628] LustreError: 166824:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5321.645031] LustreError: 166014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789055850 with bad export cookie 7525882113148362592 [ 5321.652584] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5321.665720] LustreError: 166014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5321.951404] Lustre: server umount lustre-MDT0001 complete [ 5334.413565] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:57:42 (1789055862) [ 5352.751477] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 5366.359893] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 5406.460025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.689709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5406.711824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5406.788316] Lustre: lustre-MDT0000: new disk, initializing [ 5406.888236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.929889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5411.726815] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5422.739382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5423.104380] Lustre: 172751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5423.561357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5428.920308] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5438.036035] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5443.858872] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5450.550691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5450.890575] Lustre: lustre-OST0000: new disk, initializing [ 5450.895090] Lustre: Skipped 1 previous similar message [ 5450.905478] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5450.927302] Lustre: Skipped 2 previous similar messages [ 5450.938571] Lustre: 174728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5456.946988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5456.967024] Lustre: Skipped 1 previous similar message [ 5456.980571] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5457.132935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5457.609464] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5468.371487] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5477.859704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5477.863542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.878067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5477.891810] Lustre: Skipped 2 previous similar messages [ 5477.901907] Lustre: Skipped 4 previous similar messages [ 5482.726149] Lustre: server umount lustre-OST0000 complete [ 5490.150960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5490.175936] LustreError: Skipped 1 previous similar message [ 5494.246891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5494.278709] Lustre: Skipped 1 previous similar message [ 5495.128692] Lustre: server umount lustre-MDT0000 complete [ 5498.413408] LustreError: 172759:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5498.446890] LustreError: 172759:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5500.892183] LustreError: 172743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056030 with bad export cookie 7525882113148363474 [ 5500.920115] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5501.544348] Lustre: server umount lustre-MDT0001 complete [ 5514.397280] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:00:42 (1789056042) [ 5533.805978] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 5546.052480] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 5589.036330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5589.211199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5589.235784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5589.297228] Lustre: lustre-MDT0000: new disk, initializing [ 5589.361704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5589.364858] Lustre: Skipped 2 previous similar messages [ 5589.372269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5594.549831] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5606.218623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.421735] Lustre: 179067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5606.699869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5613.986116] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5626.033492] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5632.303584] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5640.333010] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5640.636527] Lustre: 181042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5647.530862] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5649.000608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5649.169937] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5656.768739] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5669.860206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.864414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5669.905700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5669.909592] LustreError: Skipped 1 previous similar message [ 5669.928419] Lustre: Skipped 5 previous similar messages [ 5671.394082] Lustre: server umount lustre-OST0000 complete [ 5678.575209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5682.963416] Lustre: server umount lustre-MDT0000 complete [ 5685.216621] LustreError: 179077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.227250] LustreError: 179077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5687.323331] LustreError: 179057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056216 with bad export cookie 7525882113148364559 [ 5687.324837] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5687.331926] LustreError: 179057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5687.694808] Lustre: server umount lustre-MDT0001 complete [ 5699.171037] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 5702.515773] Key type lgssc unregistered [ 5702.944939] LNet: 182900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5702.959286] LNetError: 182900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5702.992073] LNet: Removed LNI 192.168.201.157@tcp [ 5704.014761] Key type .llcrypt unregistered [ 5704.016671] Key type ._llcrypt unregistered [ 5731.744169] Key type ._llcrypt registered [ 5731.749484] Key type .llcrypt registered [ 5731.983377] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 5734.092378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5734.204168] alg: No test for adler32 (adler32-zlib) [ 5735.343727] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 5735.513184] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 5737.127487] Key type lgssc registered [ 5738.382599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5748.505906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5748.530820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.872632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5756.817593] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5760.482615] LustreError: 184314:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5760.512302] LustreError: 184314:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5764.774593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5765.173096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5768.809542] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5777.924652] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5785.751862] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5795.666383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5795.817814] LustreError: 186137:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5796.485872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5801.986667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5803.624740] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5814.617847] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5835.746257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5835.752521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5835.775439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5840.893436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5840.903376] Lustre: Skipped 2 previous similar messages [ 5841.163196] Lustre: server umount lustre-OST0000 complete [ 5845.984234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5845.997121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5846.002215] Lustre: Skipped 2 previous similar messages [ 5851.119400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5851.131195] Lustre: Skipped 2 previous similar messages [ 5853.866817] Lustre: server umount lustre-MDT0000 complete [ 5856.225916] LustreError: 184313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.253946] LustreError: 184313:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5857.571693] LustreError: 186159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056386 with bad export cookie 4916068596380252695 [ 5857.577095] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5857.589980] LustreError: 186159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5858.113496] Lustre: server umount lustre-MDT0001 complete [ 5870.599311] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 5873.735694] Key type lgssc unregistered [ 5874.115937] LNet: 187872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5874.121729] LNetError: 187872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5874.141227] LNet: Removed LNI 192.168.201.157@tcp [ 5875.397610] Key type .llcrypt unregistered [ 5875.399938] Key type ._llcrypt unregistered [ 5903.820906] Key type ._llcrypt registered [ 5903.828036] Key type .llcrypt registered [ 5904.018219] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 5905.066107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5905.085196] alg: No test for adler32 (adler32-zlib) [ 5906.053818] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 5906.239773] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 5907.899248] Key type lgssc registered [ 5909.699875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5919.769909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5919.829815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5922.010588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5928.363698] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5931.517325] LustreError: 189285:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.562068] LustreError: 189285:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5936.613217] LustreError: 189286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.445770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.767154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5944.078481] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5951.852877] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5957.547490] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5966.443064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5966.675368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5974.747907] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 5988.759719] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5991.288600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6000.200875] Lustre: lustre-OST0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 6001.634987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6001.646833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.661025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6005.320208] Lustre: lustre-OST0000: Not available for connect from 192.168.201.57@tcp (stopping) [ 6005.333647] Lustre: Skipped 1 previous similar message [ 6005.491681] Lustre: server umount lustre-OST0000 complete [ 6013.347971] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff989058ce4000.ost_server_uuid 50 [ 6015.325166] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6035.936496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6035.946378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6035.964879] Lustre: Skipped 1 previous similar message [ 6035.973816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6038.250864] Lustre: server umount lustre-MDT0000 complete [ 6041.568619] LustreError: 189286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6041.574940] LustreError: 189286:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6042.819427] LustreError: 189265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056571 with bad export cookie 7938942393305989142 [ 6042.833518] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6042.848759] LustreError: 189265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6043.458117] Lustre: server umount lustre-MDT0001 complete [ 6053.389901] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:09:42 (1789056582) [ 6061.266517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.531336] LustreError: 193206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.584875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6065.923104] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6066.669353] LustreError: 193207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.197287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.188942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6083.035856] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6095.247489] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6105.506505] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6114.921185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6115.312692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6122.729782] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6130.830881] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6131.705282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6138.570529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.903855] Lustre: lustre-OST0001: new disk, initializing [ 6138.910933] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6138.926224] Lustre: 196051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6139.120316] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6146.053954] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6147.265456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6147.281365] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6147.411114] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6156.871365] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6162.404503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.408200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.436852] Lustre: Skipped 1 previous similar message [ 6162.440939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6162.443786] Lustre: Skipped 2 previous similar messages [ 6166.143799] Lustre: server umount lustre-OST0000 complete [ 6167.601961] LustreError: 195031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.641526] LustreError: 195031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6173.801584] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6174.024665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6180.268285] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6180.560546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6188.088075] LustreError: 196466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.127090] LustreError: 196466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6198.752371] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6198.767293] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.793823] Lustre: Skipped 1 previous similar message [ 6199.062102] Lustre: server umount lustre-OST0001 complete [ 6203.874669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6203.877483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6203.889858] Lustre: Skipped 2 previous similar messages [ 6203.897455] Lustre: Skipped 2 previous similar messages [ 6210.142651] Lustre: server umount lustre-MDT0000 complete [ 6215.273981] LustreError: 193188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056744 with bad export cookie 7938942393305990213 [ 6215.285245] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6215.305925] LustreError: 193188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6216.137606] Lustre: server umount lustre-MDT0001 complete [ 6248.307351] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:12:57 (1789056777) [ 6254.980238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.329877] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6255.403755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6255.706638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6261.371498] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6275.060765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6275.205092] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6275.251688] Lustre: 199488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6281.106121] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6289.545225] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6295.299113] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6303.822572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6303.972155] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6311.728292] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6313.661490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6319.369641] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6327.093504] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6327.199866] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6327.514967] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6327.528250] Lustre: Skipped 2 previous similar messages [ 6333.688709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6335.754401] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6344.065386] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6349.283458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6349.293868] LustreError: Skipped 1 previous similar message [ 6349.314880] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6349.340342] Lustre: Skipped 1 previous similar message [ 6349.346136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6349.372917] Lustre: Skipped 4 previous similar messages [ 6353.162567] Lustre: server umount lustre-OST0000 complete [ 6354.436127] LustreError: 201340:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6354.488897] LustreError: 201340:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 6374.885956] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6374.894684] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6377.022988] Lustre: server umount lustre-OST0001 complete [ 6383.083507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6383.106569] Lustre: Skipped 3 previous similar messages [ 6388.107993] Lustre: server umount lustre-MDT0000 complete [ 6392.847309] LustreError: 199480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789056922 with bad export cookie 7938942393305991620 [ 6392.850965] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6392.853065] LustreError: 199480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6393.115910] Lustre: server umount lustre-MDT0001 complete [ 6422.321847] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:15:51 (1789056951) [ 6440.285767] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 6452.099874] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 6489.435169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6489.691504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6489.733632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6489.902544] Lustre: lustre-MDT0000: new disk, initializing [ 6490.016869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6490.069220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6494.846846] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6506.236320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6506.387658] Lustre: 207591:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6506.433676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6506.437502] Lustre: Skipped 1 previous similar message [ 6506.519260] Lustre: lustre-MDT0001: new disk, initializing [ 6506.676729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6506.695778] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6512.307380] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6522.445370] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6527.835931] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6534.980723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6535.322604] Lustre: lustre-OST0000: new disk, initializing [ 6535.325657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6535.331356] Lustre: 209560:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6543.575924] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6544.486572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6544.500125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6544.625141] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6554.218447] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6560.004404] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6560.481314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6567.806781] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6568.257827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6570.475825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6570.476942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.482840] LustreError: Skipped 2 previous similar messages [ 6570.496623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6570.517124] Lustre: Skipped 4 previous similar messages [ 6570.572920] Lustre: Skipped 3 previous similar messages [ 6576.976669] Lustre: server umount lustre-OST0000 complete [ 6589.508062] Lustre: server umount lustre-MDT0000 complete [ 6590.950326] LustreError: 207598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6590.959570] LustreError: 207598:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 6593.702462] LustreError: 207583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057122 with bad export cookie 7938942393305993405 [ 6593.708652] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6593.713855] LustreError: 207583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6594.177730] Lustre: server umount lustre-MDT0001 complete [ 6605.430108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.623713] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6621.054975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.012891] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6633.755282] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6639.763643] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6650.325995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6662.359213] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6671.918623] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6677.804280] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6678.051744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6683.570330] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6683.729544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6687.203335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6687.218646] LustreError: Skipped 1 previous similar message [ 6687.222536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6687.228254] Lustre: Skipped 3 previous similar messages [ 6692.304852] Lustre: server umount lustre-OST0000 complete [ 6703.152466] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6708.595248] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6737.573866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6744.430570] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6751.543343] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6752.178516] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6752.196412] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6752.196504] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6756.204212] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6762.472349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6762.474049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6762.487352] LustreError: Skipped 1 previous similar message [ 6762.504777] Lustre: Skipped 8 previous similar messages [ 6766.601232] Lustre: server umount lustre-OST0000 complete [ 6776.846764] Lustre: server umount lustre-MDT0000 complete [ 6777.843992] LustreError: 211775:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.868829] LustreError: 211775:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6781.364267] LustreError: 212517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057310 with bad export cookie 7938942393305994280 [ 6781.367543] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6781.393718] LustreError: 212517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6781.812627] Lustre: server umount lustre-MDT0001 complete [ 6789.060369] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:21:58 (1789057318) [ 6794.142586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6794.688207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6794.697450] Lustre: Skipped 6 previous similar messages [ 6799.015658] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6809.133209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.304678] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6826.122288] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6832.524853] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6840.854346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6848.403393] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6855.627240] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6861.208033] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6867.191541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 6875.044586] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6875.456253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6882.806575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.943988] Lustre: lustre-OST0001: new disk, initializing [ 6882.946693] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6882.950414] Lustre: 221157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6888.359324] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6890.229931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6890.255356] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6890.431527] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6898.766313] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6905.740450] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6906.284821] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6913.280648] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6913.771699] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6921.202161] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6921.214052] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6921.234135] LustreError: Skipped 1 previous similar message [ 6921.269417] Lustre: Skipped 5 previous similar messages [ 6923.719584] Lustre: server umount lustre-OST0001 complete [ 6935.177669] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6935.647963] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6942.141660] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6942.460293] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6972.182785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6977.513121] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6977.521032] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6977.528996] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6977.532815] Lustre: Skipped 2 previous similar messages [ 6978.588746] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 6985.695484] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6987.044448] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6995.670932] Lustre: server umount lustre-OST0001 complete [ 7018.488367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7018.509263] Lustre: Skipped 9 previous similar messages [ 7026.143780] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7026.394468] Lustre: server umount lustre-OST0000 complete [ 7036.573715] Lustre: server umount lustre-MDT0000 complete [ 7038.947784] LustreError: 217750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.953847] LustreError: 217750:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 7039.602757] LustreError: 217730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057568 with bad export cookie 7938942393305995253 [ 7039.605858] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7039.609780] LustreError: 217730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7046.587205] Lustre: server umount lustre-MDT0001 complete [ 7081.732574] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:26:49 (1789057609) [ 7091.666648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7092.165406] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7092.232227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7097.983448] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7107.635052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7107.736362] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7107.803047] Lustre: 226405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7114.039534] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7122.262971] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7127.574681] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7135.291917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7135.496530] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7142.085345] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7151.281543] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7160.871166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7160.922331] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7167.880304] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7175.677886] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7177.130961] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7182.369901] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7182.731735] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7188.053679] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7188.215800] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7194.212621] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff989058ce1000.ost_server_uuid 50 [ 7195.576852] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff989058ce1000.ost_server_uuid in IDLE state after 0 sec [ 7196.528728] Lustre: Permanently deactivating lustre-OST0001 [ 7196.538874] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7200.966645] Lustre: Permanently reactivating lustre-OST0001 [ 7200.980057] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7200.985203] Lustre: Skipped 2 previous similar messages [ 7206.367942] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7206.374331] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.393144] LustreError: Skipped 3 previous similar messages [ 7206.412253] Lustre: Skipped 7 previous similar messages [ 7211.584559] Lustre: server umount lustre-OST0001 complete [ 7221.956207] Lustre: server umount lustre-OST0000 complete [ 7240.159125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7240.383784] Lustre: server umount lustre-MDT0000 complete [ 7245.225903] LustreError: 228268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057774 with bad export cookie 7938942393305996534 [ 7245.235761] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7245.242043] LustreError: 228268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7245.797104] Lustre: server umount lustre-MDT0001 complete [ 7280.486430] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:30:09 (1789057809) [ 7287.910369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.200686] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7288.261600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7288.281109] Lustre: Skipped 2 previous similar messages [ 7292.917028] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7304.715637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7305.024614] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7305.096382] Lustre: 233104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7310.575336] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7319.005805] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7324.044394] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7330.621848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7330.871112] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7330.899447] Lustre: Permanently deactivating lustre-OST0000 [ 7330.926129] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7331.203123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7331.211503] Lustre: Skipped 10 previous similar messages [ 7339.057693] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7339.285666] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7346.377777] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7354.269657] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7354.402306] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7361.778908] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7363.929893] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7372.050601] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7374.992682] Lustre: Permanently reactivating lustre-OST0000 [ 7375.005144] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7375.008479] Lustre: Skipped 2 previous similar messages [ 7384.625784] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 45 seconds. I think it's dead, and I am evicting it. exp 0000000018f8e59f, cur 1789057913 deadline 1789057898 last 1789057868 [ 7385.348178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7394.972795] Lustre: server umount lustre-OST0001 complete [ 7404.407589] Lustre: server umount lustre-OST0000 complete [ 7415.883792] Lustre: server umount lustre-MDT0000 complete [ 7419.751592] LustreError: 233095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789057948 with bad export cookie 7938942393305998207 [ 7419.763903] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7419.769416] LustreError: 233095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7420.325782] Lustre: server umount lustre-MDT0001 complete [ 7432.804914] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 7435.465943] Key type lgssc unregistered [ 7435.803187] LNet: 237984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7435.832042] LNetError: 237984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7436.901878] LNet: Removed LNI 192.168.201.157@tcp [ 7437.839057] Key type .llcrypt unregistered [ 7437.841048] Key type ._llcrypt unregistered [ 7461.337639] Key type ._llcrypt registered [ 7461.339929] Key type .llcrypt registered [ 7461.438039] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7463.124286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7463.153567] alg: No test for adler32 (adler32-zlib) [ 7464.238913] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 7464.568669] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 7466.247612] Key type lgssc registered [ 7467.485055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7477.446914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7477.477677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7478.882223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.802378] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7489.001240] LustreError: 239446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7489.041380] LustreError: 239446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7492.202343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.502564] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7492.545097] Lustre: 239436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7492.553422] Lustre: Permanently deactivating lustre-MDT0001 [ 7492.567815] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7492.700547] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7492.711205] LustreError: 239477:0:(osp_object.c:623:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7492.731762] LustreError: 239477:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7492.758383] LustreError: 239477:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 1, failed: rc = -108 [ 7492.903635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7499.764948] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7509.767445] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7515.712498] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7522.668245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.084300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.138094] LustreError: 241290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7524.169467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7529.451279] LustreError: 241290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7529.688880] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7534.570253] LustreError: 241370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7537.502129] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7538.669765] LustreError: 241370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7538.700915] LustreError: 241370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7544.857636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7545.261782] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7550.465308] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7550.475933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7552.226551] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7558.117393] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7562.099721] Lustre: Permanently reactivating lustre-MDT0001 [ 7562.115247] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7562.120494] Lustre: Skipped 2 previous similar messages [ 7568.457888] Lustre: lustre-MDT0001: Client 6e2c390f-c880-47e0-a71e-5511d5398b2d (at 192.168.201.57@tcp) reconnecting [ 7575.191328] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7576.914755] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7583.589933] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7583.844586] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7589.370780] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7589.598744] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7592.412742] Lustre: Permanently deactivating lustre-MDT0001 [ 7592.440389] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7592.447419] Lustre: Skipped 2 previous similar messages [ 7601.721684] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7611.882109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7611.893655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7611.914704] Lustre: Skipped 3 previous similar messages [ 7611.925010] Lustre: Skipped 3 previous similar messages [ 7616.992854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7617.011588] Lustre: Skipped 3 previous similar messages [ 7618.121041] Lustre: server umount lustre-MDT0000 complete [ 7622.119991] LustreError: 240556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7622.155914] LustreError: 240556:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7622.408234] LustreError: 239426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058151 with bad export cookie 4380863124081266401 [ 7622.413667] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7622.419784] LustreError: 239426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7623.010342] Lustre: server umount lustre-MDT0001 complete [ 7629.189797] Lustre: server umount lustre-OST0000 complete [ 7641.410104] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:36:10 (1789058170) [ 7651.887868] Lustre: server umount lustre-OST0001 complete [ 7658.725462] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 7669.973792] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7707.197716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7707.669466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7707.681897] Lustre: Skipped 2 previous similar messages [ 7707.728660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7707.845818] Lustre: lustre-MDT0000: new disk, initializing [ 7707.910770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7707.927538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7713.626165] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7725.592234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7725.796939] Lustre: 247757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7725.904507] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7725.913896] Lustre: Skipped 1 previous similar message [ 7726.085497] Lustre: lustre-MDT0001: new disk, initializing [ 7726.277595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7726.351135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7726.378104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7730.712542] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7740.390981] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7746.591706] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7753.733615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7754.030242] Lustre: lustre-OST0000: new disk, initializing [ 7754.035396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7754.050406] Lustre: 249722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7754.156695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7761.304644] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7763.566837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7763.575984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7763.711312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7771.322457] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7773.839428] LustreError: 247761:0:(mdt_handler.c:3292:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7782.720755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7782.921197] Lustre: lustre-OST0001: new disk, initializing [ 7782.929162] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7782.944318] Lustre: 250840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7783.937856] LustreError: 247761:0:(mdt_handler.c:3292:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7784.002538] LustreError: 247763:0:(mdt_handler.c:3292:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7789.693829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7789.714992] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7789.911383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7790.002489] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7794.107701] LustreError: 247763:0:(mdt_handler.c:3292:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7801.592102] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7804.543319] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7810.528469] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7810.539840] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7810.546683] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7810.548707] Lustre: Skipped 1 previous similar message [ 7810.575826] Lustre: Skipped 1 previous similar message [ 7812.878487] Lustre: server umount lustre-OST0001 complete [ 7815.650269] LustreError: 249707:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.682030] LustreError: 249707:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7830.496870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7830.497685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7830.501262] LustreError: Skipped 2 previous similar messages [ 7830.502669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7830.510293] Lustre: Skipped 1 previous similar message [ 7835.407197] Lustre: server umount lustre-OST0000 complete [ 7841.250000] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7841.254071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7841.267075] Lustre: Skipped 1 previous similar message [ 7841.273844] Lustre: Skipped 3 previous similar messages [ 7846.322931] Lustre: server umount lustre-MDT0000 complete [ 7851.489391] LustreError: 247766:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7851.515655] LustreError: 247766:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 7853.306459] LustreError: 247748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058382 with bad export cookie 4380863124081268998 [ 7853.309955] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7853.332055] LustreError: 247748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7859.925734] Lustre: server umount lustre-MDT0001 complete [ 7874.038745] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 7877.217899] Key type lgssc unregistered [ 7877.415962] LNet: 252873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7877.419634] LNetError: 252873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7877.431761] LNet: Removed LNI 192.168.201.157@tcp [ 7878.300119] Key type .llcrypt unregistered [ 7878.301354] Key type ._llcrypt unregistered [ 7926.980687] Key type ._llcrypt registered [ 7926.984618] Key type .llcrypt registered [ 7927.174935] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 7928.891099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7928.937066] alg: No test for adler32 (adler32-zlib) [ 7929.962885] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 7930.296062] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 7932.119386] Key type lgssc registered [ 7933.574797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7941.757469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7941.793201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7943.150753] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7943.176961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7943.386883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7949.561116] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7959.624106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7959.793718] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7959.832353] Lustre: 255081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7960.030459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7964.573715] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 7972.456731] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7979.351984] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7988.398972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7988.694697] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7989.027476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7995.550429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7996.541079] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8004.779300] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8056.800933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8056.805732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8056.815947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8057.316843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8057.329708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8057.330191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8060.845387] Lustre: server umount lustre-OST0000 complete [ 8064.760294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8075.232966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8075.602217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8077.717074] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8077.723532] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8082.882110] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8082.920781] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8084.589541] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8097.521801] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8144.356609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8144.362855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8144.393990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8144.427092] Lustre: Skipped 2 previous similar messages [ 8146.845922] Lustre: server umount lustre-OST0000 complete [ 8154.593299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.602521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8154.621293] Lustre: Skipped 1 previous similar message [ 8154.622035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8158.122546] Lustre: server umount lustre-MDT0000 complete [ 8163.611497] LustreError: 257666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058692 with bad export cookie 5887708206855278015 [ 8163.618036] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8163.648619] LustreError: 257666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8164.296052] Lustre: server umount lustre-MDT0001 complete [ 8177.612202] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 8180.563687] Key type lgssc unregistered [ 8180.959534] LNet: 260502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8180.973916] LNetError: 260502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8180.996199] LNet: Removed LNI 192.168.201.157@tcp [ 8181.963194] Key type .llcrypt unregistered [ 8181.966790] Key type ._llcrypt unregistered [ 8210.734621] Key type ._llcrypt registered [ 8210.737064] Key type .llcrypt registered [ 8210.838741] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8212.670494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8212.742677] alg: No test for adler32 (adler32-zlib) [ 8213.944474] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 8214.196662] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 8215.879309] Key type lgssc registered [ 8217.880879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8229.431461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8229.480049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8231.361222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8238.103900] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8241.640539] LustreError: 261920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8241.670186] LustreError: 261920:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8246.756024] LustreError: 261919:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8247.411334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.854660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8253.638772] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8263.913282] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8271.140744] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8279.506926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8280.035561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8285.768386] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8296.554347] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8301.569814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8328.160654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8328.169124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8328.198882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8332.260122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8332.260596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8332.261013] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8332.309135] Lustre: Skipped 1 previous similar message [ 8333.925090] Lustre: server umount lustre-OST0000 complete [ 8339.936396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8339.952625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8339.983364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8342.500494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8342.504134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8342.520083] Lustre: Skipped 1 previous similar message [ 8344.850206] Lustre: server umount lustre-MDT0000 complete [ 8347.621819] LustreError: 261938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8347.670269] LustreError: 261938:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8349.403635] LustreError: 261901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789058878 with bad export cookie 12337120141563750525 [ 8349.411219] LustreError: 261901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8349.416758] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8349.863581] Lustre: server umount lustre-MDT0001 complete [ 8363.689790] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 8367.194699] Key type lgssc unregistered [ 8367.501580] LNet: 266232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8367.514773] LNetError: 266232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8368.554502] LNet: Removed LNI 192.168.201.157@tcp [ 8369.721519] Key type .llcrypt unregistered [ 8369.725899] Key type ._llcrypt unregistered [ 8390.182846] Key type ._llcrypt registered [ 8390.187910] Key type .llcrypt registered [ 8390.342518] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8391.688273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8391.800684] alg: No test for adler32 (adler32-zlib) [ 8392.978058] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 8393.306055] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 8394.975271] Key type lgssc registered [ 8396.079777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8404.574682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8404.602620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8405.936438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8410.385704] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8416.235788] LustreError: 267454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.268341] LustreError: 267454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8420.912665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8421.423578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.397395] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8438.158798] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8445.843737] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8454.581255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8455.064199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8462.536683] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8471.972350] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8476.592182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8492.001782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8492.003274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.027017] LustreError: Skipped 1 previous similar message [ 8492.033952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8492.072610] Lustre: Skipped 1 previous similar message [ 8494.268265] Lustre: server umount lustre-OST0000 complete [ 8502.241953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8502.261519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8502.269795] Lustre: Skipped 1 previous similar message [ 8502.278927] Lustre: Skipped 3 previous similar messages [ 8504.938131] Lustre: server umount lustre-MDT0000 complete [ 8507.364303] LustreError: 267449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8507.389708] LustreError: 267449:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8508.874371] LustreError: 267435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059038 with bad export cookie 10583308194485925495 [ 8508.879069] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8508.880115] LustreError: 267435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8509.135113] Lustre: server umount lustre-MDT0001 complete [ 8525.113210] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 8529.138324] Key type lgssc unregistered [ 8529.445078] LNet: 271131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8529.516400] LNetError: 271131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8529.564226] LNet: Removed LNI 192.168.201.157@tcp [ 8530.975950] Key type .llcrypt unregistered [ 8530.978579] Key type ._llcrypt unregistered [ 8564.121258] Key type ._llcrypt registered [ 8564.127895] Key type .llcrypt registered [ 8564.349518] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8565.929752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8566.160799] alg: No test for adler32 (adler32-zlib) [ 8567.554775] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 8568.039641] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 8569.935263] Key type lgssc registered [ 8571.612269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8581.778452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8581.848071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.550666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8589.751788] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8592.929167] LustreError: 272547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8592.944345] LustreError: 272547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8598.003624] LustreError: 272546:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8601.440206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8602.189417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8608.735748] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8619.005734] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8629.202551] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8640.219660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8640.524757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8648.511564] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8652.848280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8659.474320] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8688.622402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8688.626851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.629401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8688.630813] LustreError: Skipped 1 previous similar message [ 8688.663048] Lustre: Skipped 1 previous similar message [ 8691.784647] Lustre: server umount lustre-OST0000 complete [ 8698.851263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8698.856657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8698.874608] Lustre: Skipped 1 previous similar message [ 8698.892625] Lustre: Skipped 2 previous similar messages [ 8703.854415] Lustre: server umount lustre-MDT0000 complete [ 8703.969129] LustreError: 272547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8704.026777] LustreError: 272547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8708.065075] LustreError: 272540:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8708.103668] LustreError: 272540:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8708.752453] LustreError: 272526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059237 with bad export cookie 11396022987729670021 [ 8708.766357] LustreError: 272526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8708.767738] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8709.224885] Lustre: server umount lustre-MDT0001 complete [ 8722.213452] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 8725.344886] Key type lgssc unregistered [ 8725.694225] LNet: 276855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8725.704476] LNetError: 276855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8725.728088] LNet: Removed LNI 192.168.201.157@tcp [ 8726.692772] Key type .llcrypt unregistered [ 8726.695839] Key type ._llcrypt unregistered [ 8747.849432] Key type ._llcrypt registered [ 8747.851808] Key type .llcrypt registered [ 8747.936252] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8749.566581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8749.601053] alg: No test for adler32 (adler32-zlib) [ 8750.720320] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 8750.991353] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 8752.712372] Key type lgssc registered [ 8754.134622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8762.437532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8762.471339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.142774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8769.068726] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8774.638345] LustreError: 278089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8774.685545] LustreError: 278089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8779.665675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.758812] LustreError: 278091:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8780.100532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8785.709842] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8795.131226] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8801.511609] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8809.290068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8809.786088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8818.813629] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8828.960176] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8834.479165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8849.902357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.913545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8849.920591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8853.697886] Lustre: server umount lustre-OST0000 complete [ 8860.134206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.144274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8860.159102] Lustre: Skipped 2 previous similar messages [ 8860.167550] Lustre: Skipped 1 previous similar message [ 8864.569991] Lustre: server umount lustre-MDT0000 complete [ 8865.252730] LustreError: 278098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8865.269696] LustreError: 278098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8868.447360] LustreError: 278054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059397 with bad export cookie 3998534617839337819 [ 8868.454154] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8868.459944] LustreError: 278054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8868.881222] Lustre: server umount lustre-MDT0001 complete [ 8880.133875] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 8883.217355] Key type lgssc unregistered [ 8883.508345] LNet: 281803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8883.522170] LNetError: 281803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8883.535302] LNet: Removed LNI 192.168.201.157@tcp [ 8884.482348] Key type .llcrypt unregistered [ 8884.486762] Key type ._llcrypt unregistered [ 8896.401657] Key type ._llcrypt registered [ 8896.403737] Key type .llcrypt registered [ 8896.501380] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8914.135863] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 8930.303534] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 8931.218184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8931.489706] alg: No test for adler32 (adler32-zlib) [ 8932.675705] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 8932.943111] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 8934.671238] Key type lgssc registered [ 8935.562745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8975.996449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8976.050899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8977.370497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8977.397862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8977.465593] Lustre: lustre-MDT0000: new disk, initializing [ 8977.538552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8977.558411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8982.297414] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 8993.710848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8993.916880] Lustre: 285354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8993.981482] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8993.985759] Lustre: Skipped 1 previous similar message [ 8994.065624] Lustre: lustre-MDT0001: new disk, initializing [ 8994.157302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8994.241298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8994.269694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9000.258679] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9013.032416] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9019.649828] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9027.576957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9027.914384] Lustre: lustre-OST0000: new disk, initializing [ 9027.922858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9027.937383] Lustre: 287328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9028.132430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9035.266545] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9037.412446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9037.418752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9037.542377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9047.664863] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9054.747373] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9055.072475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9060.367676] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9060.826954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9063.399133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9063.403760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9063.436587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9063.456527] Lustre: Skipped 1 previous similar message [ 9068.531994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9068.552995] Lustre: Skipped 2 previous similar messages [ 9073.640252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9073.653267] Lustre: Skipped 1 previous similar message [ 9077.217752] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9077.686599] Lustre: server umount lustre-OST0000 complete [ 9083.874796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9083.895557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9083.911041] Lustre: Skipped 1 previous similar message [ 9088.993455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9089.004529] Lustre: Skipped 1 previous similar message [ 9089.940800] Lustre: server umount lustre-MDT0000 complete [ 9095.242058] LustreError: 287327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059624 with bad export cookie 769753838606997574 [ 9095.247867] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9095.253158] LustreError: 287327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9095.645644] Lustre: server umount lustre-MDT0001 complete [ 9107.721795] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:00:36 (1789059636) [ 9116.883609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9117.496750] LustreError: 289743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9117.527737] LustreError: 289743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9117.584153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9122.793149] LustreError: 289744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9124.136468] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9126.885347] LustreError: 289743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9132.011708] LustreError: 289744:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9135.583399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.109175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9140.219770] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9148.747140] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9156.715686] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9164.406429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9164.901856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9173.448386] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9180.267150] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9205.560640] Lustre: server umount lustre-OST0000 complete [ 9212.899063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9212.915908] LustreError: Skipped 1 previous similar message [ 9212.919832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.942037] Lustre: Skipped 1 previous similar message [ 9212.956690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.777143] Lustre: server umount lustre-MDT0000 complete [ 9223.649671] Lustre: 283579:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789059736/real 1789059736] req@000000003c3374d0 x1875964814509696/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789059752 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9223.653995] LustreError: 289743:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9223.709801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9223.709810] Lustre: Skipped 1 previous similar message [ 9223.772931] LustreError: 289743:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9224.227837] Lustre: 283578:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789059737/real 1789059737] req@00000000b572d1e4 x1875964814510720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789059753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9225.262943] LustreError: 289724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789059754 with bad export cookie 769753838606998456 [ 9225.273065] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9225.797757] Lustre: server umount lustre-MDT0001 complete [ 9239.996843] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 9243.520754] Key type lgssc unregistered [ 9244.020759] LNet: 293321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9244.026150] LNetError: 293321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9244.044924] LNet: Removed LNI 192.168.201.157@tcp [ 9244.964420] Key type .llcrypt unregistered [ 9244.966967] Key type ._llcrypt unregistered [ 9276.008956] Key type ._llcrypt registered [ 9276.011745] Key type .llcrypt registered [ 9276.185866] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 9294.278974] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 9295.771566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9295.809453] alg: No test for adler32 (adler32-zlib) [ 9296.891122] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 9297.163337] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 9298.951614] Key type lgssc registered [ 9300.449710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9350.453976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9350.499094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9351.784411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9351.804643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9351.857272] Lustre: lustre-MDT0000: new disk, initializing [ 9351.913107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9351.926216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9358.734371] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9373.631336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9374.022989] Lustre: 297141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9374.196808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9374.202421] Lustre: Skipped 1 previous similar message [ 9374.305740] Lustre: lustre-MDT0001: new disk, initializing [ 9374.472164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9374.619608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9374.649326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9381.015236] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9391.771600] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9398.327213] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9406.149503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9406.616478] Lustre: lustre-OST2710: new disk, initializing [ 9406.621195] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9406.642893] Lustre: 299113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9406.824389] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9414.307723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9414.330512] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9414.504674] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9414.909678] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9426.283306] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9434.488945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9434.744238] Lustre: lustre-OST03e8: new disk, initializing [ 9434.750549] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9434.768302] Lustre: 300178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9434.885466] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9440.441099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9440.461753] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9440.619760] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9442.433863] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9451.173484] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9460.354531] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9460.663733] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9466.380175] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9466.631302] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9472.363768] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9472.722819] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9478.392273] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9478.663155] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9485.624318] Lustre: server umount lustre-MDT0000 complete [ 9486.823915] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9486.835109] Lustre: Skipped 1 previous similar message [ 9489.870959] LustreError: 300177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060019 with bad export cookie 4393663974738119981 [ 9489.873320] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9489.893841] LustreError: 300177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9490.236605] Lustre: server umount lustre-MDT0001 complete [ 9504.145143] Lustre: server umount lustre-OST2710 complete [ 9508.319633] Lustre: 294939:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789060021/real 1789060021] req@00000000922be32c x1875965196179712/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789060037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9508.362583] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9508.382661] Lustre: Skipped 1 previous similar message [ 9509.174865] Lustre: server umount lustre-OST03e8 complete [ 9533.328311] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [ 9544.453474] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 9588.597535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9588.979943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9589.065330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9589.192275] Lustre: lustre-MDT0000: new disk, initializing [ 9589.382917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9589.406220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9594.693628] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9607.064356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.258737] Lustre: 305437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9607.311205] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9607.315622] Lustre: Skipped 1 previous similar message [ 9607.400825] Lustre: lustre-MDT0001: new disk, initializing [ 9607.564260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9607.614722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9607.655348] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9613.153255] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9625.302859] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9632.941637] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9641.396298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9641.726875] Lustre: lustre-OST0000: new disk, initializing [ 9641.728684] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9641.731764] Lustre: 307405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9641.803136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9648.750091] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9651.292575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9651.335569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9651.540514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9660.630971] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9666.373513] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9666.771536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9672.412593] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9672.652099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9677.281689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9677.282421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9677.296905] LustreError: Skipped 1 previous similar message [ 9677.313026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9677.318548] Lustre: Skipped 1 previous similar message [ 9677.341386] Lustre: Skipped 1 previous similar message [ 9681.059823] Lustre: server umount lustre-OST0000 complete [ 9687.522837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.540175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9692.641794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9692.652407] Lustre: Skipped 2 previous similar messages [ 9697.760655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9697.772801] Lustre: Skipped 1 previous similar message [ 9700.319260] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9700.542175] Lustre: server umount lustre-MDT0000 complete [ 9702.886331] LustreError: 305447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.911911] LustreError: 305447:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9705.606043] LustreError: 308946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060234 with bad export cookie 4393663974738121661 [ 9705.607685] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9705.616961] LustreError: 308946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9706.044963] Lustre: server umount lustre-MDT0001 complete [ 9715.394282] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:10:44 (1789060244) [ 9717.468722] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9719.720914] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:10:48 (1789060248) [ 9724.124945] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 9757.957889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.227976] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9758.262121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9758.501557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9764.798380] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9775.323079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9775.395747] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9775.417726] Lustre: 311263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9781.848956] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9805.685933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9806.097674] Lustre: Denying initial registration attempt from nid 192.168.201.157@tcp, specified as failover [ 9806.114721] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9806.122909] LustreError: 312718:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9806.130463] LustreError: 312718:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9806.138575] LustreError: 312718:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9806.262116] Lustre: server umount lustre-OST0000 complete [ 9806.263915] LustreError: 312718:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9814.512187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9814.550814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9814.558132] Lustre: Skipped 2 previous similar messages [ 9814.606775] Lustre: Skipped 1 previous similar message [ 9821.304997] Lustre: server umount lustre-MDT0000 complete [ 9824.737584] LustreError: 312336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9824.765109] LustreError: 312336:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9826.526313] LustreError: 311254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060355 with bad export cookie 4393663974738122543 [ 9826.538246] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9826.540973] LustreError: 311254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9827.147468] Lustre: server umount lustre-MDT0001 complete [ 9840.129394] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [ 9843.831604] Key type lgssc unregistered [ 9844.257866] LNet: 313840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9844.273914] LNetError: 313840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9844.296407] LNet: Removed LNI 192.168.201.157@tcp [ 9845.546911] Key type .llcrypt unregistered [ 9845.549747] Key type ._llcrypt unregistered [ 9861.528580] Key type ._llcrypt registered [ 9861.531576] Key type .llcrypt registered [ 9861.745508] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 9863.768674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9863.781196] alg: No test for adler32 (adler32-zlib) [ 9865.010910] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [ 9865.289735] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [ 9867.007373] Key type lgssc registered [ 9868.567658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9920.458458] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [ 9930.739675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9930.766958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9932.088198] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9932.109908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9932.283415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9937.003732] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9946.541827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9946.693942] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9946.729406] Lustre: 316604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9947.011486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9953.320733] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [ 9962.981869] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9969.520943] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9991.164587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9991.508953] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9991.931744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10000.243475] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10010.107614] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10015.251225] Lustre: server umount lustre-OST0000 complete [10022.369911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10022.390883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10022.402574] Lustre: Skipped 1 previous similar message [10027.490692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10027.495808] Lustre: Skipped 1 previous similar message [10030.721547] Lustre: server umount lustre-MDT0000 complete [10033.631192] Lustre: 314496:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789060546/real 1789060546] req@0000000030e86154 x1875965791733248/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789060562 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10033.652142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10033.674903] Lustre: Skipped 1 previous similar message [10035.114942] LustreError: 316595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060564 with bad export cookie 7941859676892946494 [10035.117595] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10035.125508] LustreError: 316595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10035.487147] Lustre: server umount lustre-MDT0001 complete [10048.959557] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [10052.294820] Key type lgssc unregistered [10052.675213] LNet: 320178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10052.681747] LNetError: 320178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10053.735249] LNet: Removed LNI 192.168.201.157@tcp [10054.942658] Key type .llcrypt unregistered [10054.949140] Key type ._llcrypt unregistered [10084.854915] Key type ._llcrypt registered [10084.858499] Key type .llcrypt registered [10085.000881] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10086.534545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10086.658581] alg: No test for adler32 (adler32-zlib) [10087.870402] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [10088.224754] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [10090.015399] Key type lgssc registered [10091.507661] Lustre: Echo OBD driver; http://www.lustre.org/ [10100.710564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10100.728685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10102.350920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10107.681490] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10112.484708] LustreError: 321601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10112.528891] LustreError: 321601:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10117.501752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10117.602662] LustreError: 321600:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10117.943308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10123.796185] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10132.583302] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10139.528204] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10151.029523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10151.572780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10160.704448] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10170.198573] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10182.627364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10182.633607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10182.647781] Lustre: Skipped 2 previous similar messages [10182.663241] Lustre: Skipped 2 previous similar messages [10183.766925] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.57@tcp (stopping) [10186.558406] Lustre: server umount lustre-MDT0000 complete [10187.759341] LustreError: 321596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.809755] LustreError: 321596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10191.753115] LustreError: 321582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060720 with bad export cookie 17806097999692446503 [10191.763966] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10192.865377] LustreError: 321596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10192.866093] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10192.884236] LustreError: 321596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10192.923649] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10196.904464] LustreError: 324100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.904777] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10196.926758] LustreError: 324100:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10196.936692] Lustre: Skipped 1 previous similar message [10200.998861] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10201.003102] Lustre: Skipped 1 previous similar message [10205.329515] LustreError: 321597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10205.356041] LustreError: 321597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10206.175367] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10206.524772] Lustre: server umount lustre-MDT0001 complete [10214.315384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10224.621024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.270185] LustreError: 323433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10236.302662] LustreError: 323433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10236.411552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10236.483156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10241.707596] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10251.169678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10251.563170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10256.565237] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10260.973049] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10265.498817] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10267.402275] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10274.976470] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10277.027675] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10286.561394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.575667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10286.589197] Lustre: Skipped 4 previous similar messages [10289.767503] Lustre: server umount lustre-OST0000 complete [10296.801671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10296.828227] Lustre: Skipped 2 previous similar messages [10300.861441] Lustre: server umount lustre-MDT0000 complete [10301.921588] LustreError: 325205:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10301.960912] LustreError: 325205:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10305.525854] LustreError: 321580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060834 with bad export cookie 17806097999692454126 [10305.541361] LustreError: 321580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10305.545251] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10306.044751] Lustre: server umount lustre-MDT0001 complete [10321.664035] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [10326.212478] Key type lgssc unregistered [10326.792374] LNet: 327877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10326.796501] LNetError: 327877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10326.835984] LNet: Removed LNI 192.168.201.157@tcp [10328.100938] Key type .llcrypt unregistered [10328.105254] Key type ._llcrypt unregistered [10359.091547] Key type ._llcrypt registered [10359.094595] Key type .llcrypt registered [10359.294469] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10361.668896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10361.720298] alg: No test for adler32 (adler32-zlib) [10363.000391] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [10363.423411] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [10365.167318] Key type lgssc registered [10366.661738] Lustre: Echo OBD driver; http://www.lustre.org/ [10375.895797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10375.909915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10377.533711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10383.929853] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10386.920274] LustreError: 329291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10386.941208] LustreError: 329291:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10392.038377] LustreError: 329292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.828826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10396.475652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10402.573135] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10412.078827] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10420.024171] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10427.363605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10427.370220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10427.370419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10427.388745] Lustre: Skipped 1 previous similar message [10432.488049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10432.503579] Lustre: Skipped 2 previous similar messages [10437.605619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10437.618358] Lustre: Skipped 1 previous similar message [10438.111378] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10438.331896] Lustre: server umount lustre-MDT0000 complete [10441.701885] LustreError: 329292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10441.730092] LustreError: 329292:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10443.222620] LustreError: 329274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789060972 with bad export cookie 3226297149866813722 [10443.229718] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10443.242938] LustreError: 329274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10443.559635] Lustre: server umount lustre-MDT0001 complete [10461.636136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10462.080778] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10462.115377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10462.323576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10467.605471] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10479.122727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.415356] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10479.461235] Lustre: 331884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10479.772751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10485.757357] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10494.675480] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10501.251981] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10511.863620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10512.163407] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10512.174092] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10512.181888] LustreError: 333767:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10512.193140] LustreError: 333767:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10512.203640] LustreError: 333767:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10512.259453] Lustre: server umount lustre-OST0000 complete [10512.262299] LustreError: 333767:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10523.778571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10524.206573] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10524.510906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10532.157569] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10534.613570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10540.574215] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10550.303182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10550.426870] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10550.532962] Lustre: lustre-OST0001: new disk, initializing [10550.536529] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10550.542419] Lustre: 335313:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10550.634955] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10556.627229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10556.638984] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10556.799682] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10559.316089] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10567.842755] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10572.261042] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10572.274000] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10572.304301] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10572.317266] Lustre: Skipped 1 previous similar message [10577.385128] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10577.407266] Lustre: Skipped 2 previous similar messages [10577.901905] Lustre: server umount lustre-OST0001 complete [10582.542257] LustreError: 334233:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10582.562881] LustreError: 334233:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10585.571038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10585.584342] LustreError: Skipped 1 previous similar message [10585.594611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10585.622040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10587.625505] LustreError: 335563:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10587.638466] LustreError: 335563:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10589.018510] Lustre: server umount lustre-OST0000 complete [10597.345028] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10597.354962] LustreError: Skipped 1 previous similar message [10597.359574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10597.371220] Lustre: Skipped 1 previous similar message [10599.589048] Lustre: server umount lustre-MDT0000 complete [10602.979237] LustreError: 331896:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10602.998036] LustreError: 331896:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10604.151560] LustreError: 332623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061133 with bad export cookie 3226297149866814275 [10604.158410] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10604.162537] LustreError: 332623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10604.724740] Lustre: server umount lustre-MDT0001 complete [10618.819929] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [10622.628797] Key type lgssc unregistered [10623.040330] LNet: 337271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10623.055546] LNetError: 337271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10623.077682] LNet: Removed LNI 192.168.201.157@tcp [10624.196807] Key type .llcrypt unregistered [10624.198603] Key type ._llcrypt unregistered [10709.991658] Key type ._llcrypt registered [10709.995091] Key type .llcrypt registered [10710.157801] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [10729.136654] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [10730.657835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10730.922902] alg: No test for adler32 (adler32-zlib) [10732.207507] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [10732.463183] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [10734.143185] Key type lgssc registered [10735.973717] Lustre: Echo OBD driver; http://www.lustre.org/ [10780.506679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10780.547928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10782.002757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10782.078959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10782.221817] Lustre: lustre-MDT0000: new disk, initializing [10782.329441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10782.356514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10787.762193] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10800.412807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.606951] Lustre: 342203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10800.668376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10800.673262] Lustre: Skipped 1 previous similar message [10800.774323] Lustre: lustre-MDT0001: new disk, initializing [10801.043877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10801.140441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10801.182392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10808.302050] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10819.550395] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10826.780668] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10835.139356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10835.605218] Lustre: lustre-OST0000: new disk, initializing [10835.615329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10835.627783] Lustre: 344176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10835.734576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10841.235398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10841.250393] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10841.429632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10843.777660] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10855.609380] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10860.117170] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10860.360715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10865.086829] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10865.277188] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10867.170550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10867.172986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10867.185472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10867.221969] Lustre: Skipped 1 previous similar message [10872.293308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10872.308180] Lustre: Skipped 2 previous similar messages [10873.717106] Lustre: server umount lustre-OST0000 complete [10882.533151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10882.537358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10882.546576] Lustre: Skipped 1 previous similar message [10882.570882] Lustre: Skipped 1 previous similar message [10887.651814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10887.660883] Lustre: Skipped 1 previous similar message [10892.769453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10892.782838] Lustre: Skipped 1 previous similar message [10894.304448] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10894.880852] Lustre: server umount lustre-MDT0000 complete [10899.260946] LustreError: 342197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061428 with bad export cookie 16575418389369437511 [10899.271746] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10899.273857] LustreError: 342197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10899.726632] Lustre: server umount lustre-MDT0001 complete [10909.554687] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:30:38 (1789061438) [10919.757404] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:30:48 (1789061448) [10928.933542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10929.703297] LustreError: 346933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10929.717788] LustreError: 346933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10929.750132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10934.756162] LustreError: 346934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.670052] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10938.853617] LustreError: 346933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10943.980025] LustreError: 346934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.791744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10948.678924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10954.714711] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10962.943781] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10969.344477] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10977.926903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10978.736861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10987.127902] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [10997.354735] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11011.155211] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11013.446468] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11015.040075] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11016.834241] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11019.194732] Lustre: Failing over lustre-MDT0000 [11019.598561] Lustre: server umount lustre-MDT0000 complete [11019.749930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11019.773441] LustreError: 346933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11019.782386] Lustre: Skipped 1 previous similar message [11029.032767] LustreError: 348978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11029.052455] LustreError: 348978:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [11041.247164] Lustre: 340426:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789061554/real 1789061554] req@000000009837065f x1875966700892416/t0(0) o400->MGC192.168.201.157@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789061570 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11041.287871] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11041.521965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11045.956846] LustreError: 346929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11046.005426] LustreError: 346929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [11051.516706] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe607b7346dd7c4d3 [11051.543310] Lustre: MGC192.168.201.157@tcp: Connection restored to 0@lo (at 0@lo) [11051.865809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11051.928123] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11055.146648] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11057.152113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11057.224519] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11057.305269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11058.160977] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11068.530363] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11070.646554] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11075.288557] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11077.345653] Lustre: Failing over lustre-MDT0000 [11077.605217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11077.606493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11077.615286] LustreError: Skipped 1 previous similar message [11077.637727] Lustre: Skipped 1 previous similar message [11077.873215] Lustre: server umount lustre-MDT0000 complete [11080.771609] LustreError: 346928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11080.789661] LustreError: 346928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11097.994883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11098.236197] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11098.550361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11098.598603] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11102.252673] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11103.738098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11103.748178] Lustre: Skipped 2 previous similar messages [11103.790206] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11103.861801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11104.025504] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11114.870166] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11117.200538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11124.192926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11124.219839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11124.253923] Lustre: Skipped 3 previous similar messages [11124.259460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11124.267806] Lustre: Skipped 2 previous similar messages [11130.558519] Lustre: server umount lustre-MDT0000 complete [11136.143973] LustreError: 350933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061665 with bad export cookie 16575418389369440220 [11136.156461] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11136.172819] LustreError: 350933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11137.101512] Lustre: server umount lustre-MDT0001 complete [11151.373975] Lustre: server umount lustre-OST0000 complete [11170.882915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11171.712889] LustreError: 353309:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.735256] LustreError: 353309:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [11171.812862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11177.616517] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11187.661207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.831410] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11203.812909] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11211.323601] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11219.671208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11228.247264] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11237.129974] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11241.742059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11249.121540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11249.127183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11249.140958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11249.143134] Lustre: Skipped 3 previous similar messages [11250.418685] Lustre: server umount lustre-OST0000 complete [11270.111240] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11270.362734] Lustre: server umount lustre-MDT0000 complete [11274.703993] LustreError: 353291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061803 with bad export cookie 16575418389369440969 [11274.717142] LustreError: 353291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11274.724824] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11275.595726] Lustre: server umount lustre-MDT0001 complete [11288.916680] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [11292.093193] Key type lgssc unregistered [11292.565655] LNet: 356841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11292.579038] LNetError: 356841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11292.606277] LNet: Removed LNI 192.168.201.157@tcp [11293.728312] Key type .llcrypt unregistered [11293.730068] Key type ._llcrypt unregistered [11325.496185] Key type ._llcrypt registered [11325.498923] Key type .llcrypt registered [11325.681080] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [11327.549145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11327.627081] alg: No test for adler32 (adler32-zlib) [11328.794626] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [11329.170801] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [11330.904244] Key type lgssc registered [11332.834243] Lustre: Echo OBD driver; http://www.lustre.org/ [11343.261605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11343.305865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11344.963579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11349.555214] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11355.113474] LustreError: 358358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11355.138354] LustreError: 358358:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11359.787484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.224676] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11360.428442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11365.403669] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11375.256054] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11381.482185] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11389.674385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11390.184913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11391.258551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11397.852482] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11407.136801] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11421.155573] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11421.165765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11421.167797] Lustre: Skipped 2 previous similar messages [11421.178536] Lustre: Skipped 2 previous similar messages [11423.203059] Lustre: server umount lustre-MDT0000 complete [11426.276496] LustreError: 360240:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11426.311890] LustreError: 360240:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11427.574651] LustreError: 358338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789061956 with bad export cookie 13074737947076275869 [11427.579537] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11427.593574] LustreError: 358338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11427.969952] Lustre: server umount lustre-MDT0001 complete [11444.610817] Lustre: server umount lustre-OST0000 complete [11462.263226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11462.994486] LustreError: 362073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11463.105694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11468.115305] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11468.273654] LustreError: 362074:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11468.286367] LustreError: 362074:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11473.377626] LustreError: 362073:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11477.392128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11478.014290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11483.949048] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11492.662360] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11498.591165] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11506.141600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11506.497106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11508.635262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11513.916404] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11521.809591] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11538.405808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11538.408367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11538.431249] Lustre: Skipped 2 previous similar messages [11538.445646] Lustre: Skipped 2 previous similar messages [11541.339868] Lustre: server umount lustre-MDT0000 complete [11543.526723] LustreError: 362068:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11543.551748] LustreError: 362068:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11546.869565] LustreError: 362053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062076 with bad export cookie 13074737947076276982 [11546.872322] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11546.881030] LustreError: 362053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11547.379029] Lustre: server umount lustre-MDT0001 complete [11563.480702] Lustre: server umount lustre-OST0000 complete [11582.894714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11583.413794] LustreError: 365985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11583.471470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11589.277367] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11599.836680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11606.599177] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11616.900604] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11624.108490] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11631.550877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11632.005416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11632.012208] Lustre: Skipped 1 previous similar message [11639.202729] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11648.122926] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11653.500119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11660.768369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11660.777264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11660.798896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11663.848192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11663.867647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11665.509442] Lustre: server umount lustre-OST0000 complete [11672.041510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11672.059890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11672.086388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11672.094602] Lustre: Skipped 2 previous similar messages [11677.156036] Lustre: server umount lustre-MDT0000 complete [11679.200135] LustreError: 365985:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11679.224466] LustreError: 365985:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11681.666605] LustreError: 365967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062210 with bad export cookie 13074737947076278004 [11681.692700] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11682.042438] Lustre: server umount lustre-MDT0001 complete [11696.334132] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [11699.637751] Key type lgssc unregistered [11700.165549] LNet: 369519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11700.178233] LNetError: 369519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11700.200918] LNet: Removed LNI 192.168.201.157@tcp [11701.300071] Key type .llcrypt unregistered [11701.302642] Key type ._llcrypt unregistered [11727.894657] Key type ._llcrypt registered [11727.896295] Key type .llcrypt registered [11727.981160] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [11729.850241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11729.906686] alg: No test for adler32 (adler32-zlib) [11731.127339] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [11731.448581] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [11733.344447] Key type lgssc registered [11734.616757] Lustre: Echo OBD driver; http://www.lustre.org/ [11743.592869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11743.605913] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11743.612390] LustreError: 370957:0:(osd_handler.c:8543:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11743.630981] LustreError: 370957:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11743.644987] LustreError: 370957:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11743.651855] LustreError: 370957:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11743.658934] LustreError: 370957:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11752.551081] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11752.554236] LustreError: 371356:0:(osd_handler.c:8543:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11752.564483] LustreError: 371356:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11752.568636] LustreError: 371356:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11752.572546] LustreError: 371356:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11752.575481] LustreError: 371356:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11774.461772] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [11777.919896] Key type lgssc unregistered [11778.254911] LNet: 372359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11778.260690] LNetError: 372359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11778.279780] LNet: Removed LNI 192.168.201.157@tcp [11779.102579] Key type .llcrypt unregistered [11779.105093] Key type ._llcrypt unregistered [11799.185862] Key type ._llcrypt registered [11799.190228] Key type .llcrypt registered [11799.284956] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [11817.712387] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [11819.526586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11819.614815] alg: No test for adler32 (adler32-zlib) [11820.818353] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [11821.217193] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [11823.031239] Key type lgssc registered [11824.597949] Lustre: Echo OBD driver; http://www.lustre.org/ [11866.317480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11866.342763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11867.747550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11867.811357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11867.960663] Lustre: lustre-MDT0000: new disk, initializing [11868.085941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11868.136996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11874.176234] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11890.097480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11890.362941] Lustre: 375515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11890.458764] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11890.462864] Lustre: Skipped 1 previous similar message [11890.631760] Lustre: lustre-MDT0001: new disk, initializing [11890.826713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11890.886549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11890.902493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11896.222893] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11910.496507] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11919.008438] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11929.289252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11929.714690] Lustre: lustre-OST0000: new disk, initializing [11929.724278] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11929.737761] Lustre: 377484:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11929.874431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11935.304444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11935.328081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11935.473402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11937.511885] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [11949.996624] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11955.996495] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11956.272774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11962.489330] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11962.742771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11966.444405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11966.448913] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11966.450702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11966.460126] LustreError: Skipped 1 previous similar message [11966.509813] Lustre: Skipped 1 previous similar message [11971.563597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11971.568747] Lustre: Skipped 2 previous similar messages [11976.673813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11976.687821] Lustre: Skipped 1 previous similar message [11979.232452] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11979.503288] Lustre: server umount lustre-OST0000 complete [11986.914561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11986.928655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11986.931093] Lustre: Skipped 1 previous similar message [11989.917048] Lustre: server umount lustre-MDT0000 complete [11994.503753] LustreError: 375508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062523 with bad export cookie 6619641123602240656 [11994.516347] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11994.523613] LustreError: 375508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11995.205635] Lustre: server umount lustre-MDT0001 complete [12007.883647] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12010.436637] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:48:58 (1789062538) [12019.998931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12020.937253] LustreError: 379948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.966203] LustreError: 379948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12021.076597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12026.340571] LustreError: 379949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12026.366756] LustreError: 379949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12026.812307] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12031.464396] LustreError: 379948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12036.580431] LustreError: 379949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12037.556825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12038.414216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12043.497405] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12052.372747] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12060.387641] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12069.177866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12069.611932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12076.392297] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12088.002590] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12100.298411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12100.672930] Lustre: lustre-OST0001: new disk, initializing [12100.707596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12100.731982] Lustre: 382788:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12100.945594] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12108.379336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12108.420784] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12108.547504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12108.818414] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12119.530633] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12124.133894] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12124.153106] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12124.166138] Lustre: Skipped 1 previous similar message [12124.174326] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12129.641450] Lustre: server umount lustre-OST0001 complete [12131.385896] LustreError: 381773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12139.526816] LustreError: 381773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12139.556076] LustreError: 381773:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12149.731473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.768277] Lustre: Skipped 1 previous similar message [12149.775953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12149.783696] Lustre: Skipped 5 previous similar messages [12154.604694] Lustre: server umount lustre-OST0000 complete [12159.968830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12159.990748] Lustre: Skipped 2 previous similar messages [12166.224098] Lustre: server umount lustre-MDT0000 complete [12170.211670] LustreError: 379948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12170.241145] LustreError: 379948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12171.030019] LustreError: 384024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062700 with bad export cookie 6619641123602241538 [12171.039178] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12171.042492] LustreError: 384024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12177.605595] Lustre: server umount lustre-MDT0001 complete [12191.184864] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [12194.905304] Key type lgssc unregistered [12195.342093] LNet: 384777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12195.352111] LNetError: 384777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12195.363666] LNet: Removed LNI 192.168.201.157@tcp [12196.171345] Key type .llcrypt unregistered [12196.173203] Key type ._llcrypt unregistered [12259.790690] Key type ._llcrypt registered [12259.793534] Key type .llcrypt registered [12259.972861] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [12261.631664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12261.673537] alg: No test for adler32 (adler32-zlib) [12262.763919] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [12263.067327] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [12264.775413] Key type lgssc registered [12266.158719] Lustre: Echo OBD driver; http://www.lustre.org/ [12275.032240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12275.043582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12276.405662] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12276.435539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12276.697830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12282.413990] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12292.687238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12292.866370] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12292.972347] Lustre: 387573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12293.472868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12298.858296] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12308.563028] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12314.638624] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12319.208910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12319.210594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12319.222796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12323.572045] Lustre: server umount lustre-MDT0000 complete [12324.322138] LustreError: 388569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12324.351553] LustreError: 388569:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12329.448524] LustreError: 387585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12329.482764] LustreError: 387585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12329.663083] LustreError: 387566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789062858 with bad export cookie 7951722657708767200 [12329.670730] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12329.695943] LustreError: 387566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12330.521586] Lustre: server umount lustre-MDT0001 complete [12345.259804] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:54:34 (1789062874) [12353.694088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12354.681033] LustreError: 390089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12354.788773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12360.170725] LustreError: 390090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12361.600705] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12365.281348] LustreError: 390089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12372.367914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12372.843087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12378.724661] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12387.183719] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12393.808661] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12401.140480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12401.437674] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12401.695243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12409.542754] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12418.790426] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12428.570882] Lustre: Permanently deactivating lustre-OST0000 [12428.587842] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12438.587794] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12440.194297] LustreError: 393077:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12440.218360] LustreError: 393077:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12440.233957] LustreError: 393077:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12445.843506] LustreError: 393125:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12445.855707] LustreError: 393125:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12450.803947] Lustre: server umount lustre-OST0000 complete [12454.783593] LustreError: 393375:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12454.808681] LustreError: 393375:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12460.007038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12460.018203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12460.024729] Lustre: Skipped 2 previous similar messages [12460.025110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12460.029551] Lustre: Skipped 2 previous similar messages [12463.009123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12463.019777] Lustre: Skipped 1 previous similar message [12468.195930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12468.215811] Lustre: Skipped 1 previous similar message [12472.287169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12472.692057] Lustre: server umount lustre-MDT0000 complete [12473.313874] LustreError: 390090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12473.348509] LustreError: 390090:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12477.812355] LustreError: 392781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063006 with bad export cookie 7951722657708767781 [12477.813660] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12477.824854] LustreError: 392781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12487.216536] Lustre: server umount lustre-MDT0001 complete [12497.591119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12503.829301] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12506.977913] LustreError: 394537:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12514.340918] Lustre: 394876:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.157@tcp [12517.120324] Lustre: server umount MGS complete [12535.113074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12536.219214] LustreError: 395601:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12536.261699] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12536.266443] Lustre: Skipped 1 previous similar message [12536.327750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12542.615304] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12555.360891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12555.646348] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12555.718722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12561.316744] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12570.386546] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12578.613650] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12581.923976] Lustre: Permanently reactivating lustre-OST0000 [12581.939089] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12581.950453] Lustre: Skipped 2 previous similar messages [12611.551123] LustreError: 397386:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12620.796809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12621.334817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12629.406666] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12641.402360] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12652.001731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12652.006484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12652.022144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12652.040123] Lustre: Skipped 1 previous similar message [12656.544535] Lustre: server umount lustre-OST0000 complete [12662.241345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12662.249183] Lustre: Skipped 1 previous similar message [12662.253775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12662.264812] Lustre: Skipped 1 previous similar message [12668.173554] Lustre: server umount lustre-MDT0000 complete [12672.481594] LustreError: 395596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12672.500992] LustreError: 395596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12672.920501] LustreError: 395584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063202 with bad export cookie 7951722657708768831 [12672.930312] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12672.933377] LustreError: 395584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12673.633942] Lustre: server umount lustre-MDT0001 complete [12688.048409] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [12692.110849] Key type lgssc unregistered [12692.583126] LNet: 399922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12692.597811] LNetError: 399922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12692.621246] LNet: Removed LNI 192.168.201.157@tcp [12694.226966] Key type .llcrypt unregistered [12694.231431] Key type ._llcrypt unregistered [12721.017392] Key type ._llcrypt registered [12721.022239] Key type .llcrypt registered [12721.203233] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [12744.452147] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [12746.712270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12746.789344] alg: No test for adler32 (adler32-zlib) [12748.024156] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [12748.516732] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [12750.312583] Key type lgssc registered [12752.066482] Lustre: Echo OBD driver; http://www.lustre.org/ [12800.968775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12801.010634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12802.345866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12802.391922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12802.509660] Lustre: lustre-MDT0000: new disk, initializing [12802.605026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12802.630372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12809.879632] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12822.583231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12822.767182] Lustre: 403090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12822.840411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12822.843067] Lustre: Skipped 1 previous similar message [12822.942429] Lustre: lustre-MDT0001: new disk, initializing [12823.101054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12823.158892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12823.170850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12828.263716] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12839.282410] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12848.818429] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12861.689594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12862.101284] Lustre: lustre-OST0000: new disk, initializing [12862.105546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12862.130871] Lustre: 405062:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12862.327772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12869.363487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12869.391513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12869.674064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12870.732526] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12883.808620] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12889.828845] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12890.043945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12896.132546] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12896.425437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12900.322067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12900.332026] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12900.350288] LustreError: Skipped 1 previous similar message [12900.353989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12900.407670] Lustre: Skipped 1 previous similar message [12905.371473] Lustre: server umount lustre-OST0000 complete [12910.053557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12910.060978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12910.094515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12910.096940] Lustre: Skipped 1 previous similar message [12915.681438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12915.686134] Lustre: Skipped 2 previous similar messages [12920.805760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12920.809272] Lustre: Skipped 1 previous similar message [12924.384030] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12924.730888] Lustre: server umount lustre-MDT0000 complete [12930.153687] LustreError: 405061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063459 with bad export cookie 5175767908877768310 [12930.173792] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12930.197606] LustreError: 405061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12930.833928] Lustre: server umount lustre-MDT0001 complete [12945.121657] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12948.058581] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:04:36 (1789063476) [12956.475140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12956.869466] LustreError: 407525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12956.882443] LustreError: 407525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12956.951534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12962.274415] LustreError: 407526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12962.822877] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12967.394304] LustreError: 407525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12972.514201] LustreError: 407526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12976.053503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12976.401875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12982.887577] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [12991.885395] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13000.525837] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13010.402403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13011.096573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13019.296600] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13029.248751] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13031.450958] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13035.553862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13042.144646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13042.152161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13042.167073] Lustre: Skipped 1 previous similar message [13042.169813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13047.255488] Lustre: server umount lustre-OST0000 complete [13053.412749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13053.416405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13053.439735] LustreError: Skipped 1 previous similar message [13053.442373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13053.442381] Lustre: Skipped 2 previous similar messages [13053.518886] Lustre: Skipped 2 previous similar messages [13059.483932] Lustre: server umount lustre-MDT0000 complete [13063.649569] LustreError: 408253:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13063.693808] LustreError: 408253:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13064.637250] LustreError: 407506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063593 with bad export cookie 5175767908877769199 [13064.657375] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13065.528695] Lustre: server umount lustre-MDT0001 complete [13080.769556] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [13084.674378] Key type lgssc unregistered [13085.018342] LNet: 411255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13085.031601] LNetError: 411255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13085.056775] LNet: Removed LNI 192.168.201.157@tcp [13086.678738] Key type .llcrypt unregistered [13086.681858] Key type ._llcrypt unregistered [13122.200661] Key type ._llcrypt registered [13122.202802] Key type .llcrypt registered [13122.391854] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [13123.027527] Key type .llcrypt unregistered [13123.030896] Key type ._llcrypt unregistered [13147.050079] Key type ._llcrypt registered [13147.053464] Key type .llcrypt registered [13147.179475] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [13148.473562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13148.605171] alg: No test for adler32 (adler32-zlib) [13149.712724] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [13150.124115] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [13151.887710] Key type lgssc registered [13153.730341] Lustre: Echo OBD driver; http://www.lustre.org/ [13167.094126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13167.152958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13169.050111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13175.176544] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13179.363110] LustreError: 413466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13179.407139] LustreError: 413466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13184.492626] LustreError: 413467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13186.418793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13186.741232] LustreError: 413466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13186.774185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13192.161948] LustreError: 413467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13192.217130] LustreError: 413467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13194.261240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13194.896583] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13199.349891] LustreError: 413466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13199.381369] LustreError: 413466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13203.671961] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13209.582230] LustreError: 413467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13209.619446] LustreError: 413467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13211.085112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13211.572376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13217.044408] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13230.064071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13230.074609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13230.112409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13232.100276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13232.118776] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13232.122176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13232.175468] Lustre: Skipped 1 previous similar message [13233.399564] Lustre: server umount lustre-OST0000 complete [13239.971281] Lustre: server umount lustre-MDT0000 complete [13242.338496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13242.365977] LustreError: 413461:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13242.397025] LustreError: 413461:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13244.729674] LustreError: 413447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063773 with bad export cookie 5753490004647973107 [13244.758194] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13245.246829] Lustre: server umount lustre-MDT0001 complete [13259.457063] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [13263.578711] Key type lgssc unregistered [13264.146323] LNet: 416614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13264.160719] LNetError: 416614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13264.178516] LNet: Removed LNI 192.168.201.157@tcp [13265.375674] Key type .llcrypt unregistered [13265.379626] Key type ._llcrypt unregistered [13296.136205] Key type ._llcrypt registered [13296.140565] Key type .llcrypt registered [13296.311426] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [13297.924777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13298.074948] alg: No test for adler32 (adler32-zlib) [13299.489553] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [13299.707321] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [13301.375862] Key type lgssc registered [13302.559778] Lustre: Echo OBD driver; http://www.lustre.org/ [13311.776086] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13311.797142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13327.393411] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13332.511523] LustreError: 418001:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.157@tcp: failed processing log, type 4: rc = -110 [13359.265921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13367.175361] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13376.777589] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13385.723125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13386.427194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13386.547438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13391.843484] LustreError: 418027:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13391.873098] LustreError: 418027:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13392.904216] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13393.898029] LustreError: 419038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13393.925145] LustreError: 419038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13399.020548] LustreError: 419038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13399.045069] LustreError: 419038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13402.641925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13403.269536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13408.622351] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13417.052524] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13423.860156] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13432.807729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13432.819176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13432.830795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13433.828054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13433.840852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13433.868276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13437.859798] Lustre: server umount lustre-OST0000 complete [13444.065836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13444.072793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13444.089802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13444.094235] Lustre: Skipped 2 previous similar messages [13448.121610] Lustre: server umount lustre-MDT0000 complete [13449.696551] LustreError: 419038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13449.715889] LustreError: 419038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13453.094364] LustreError: 418009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789063982 with bad export cookie 6887413651712883466 [13453.099292] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13453.114288] LustreError: 418009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13453.589638] Lustre: server umount lustre-MDT0001 complete [13470.045749] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [13475.291711] Key type lgssc unregistered [13475.746640] LNet: 421576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13475.758094] LNetError: 421576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13475.805141] LNet: Removed LNI 192.168.201.157@tcp [13476.903716] Key type .llcrypt unregistered [13476.906342] Key type ._llcrypt unregistered [13506.287234] Key type ._llcrypt registered [13506.291257] Key type .llcrypt registered [13506.434823] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [13507.727196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13507.750753] alg: No test for adler32 (adler32-zlib) [13508.752494] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [13509.037252] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [13510.703249] Key type lgssc registered [13512.096940] Lustre: Echo OBD driver; http://www.lustre.org/ [13523.264845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13523.273563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13524.702135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13530.835488] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13534.181218] LustreError: 422989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13534.218326] LustreError: 422989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13539.305798] LustreError: 422988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13544.430767] LustreError: 422989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13545.367584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13546.325869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13553.824957] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13563.384820] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13571.846970] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13580.202232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13580.563978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13587.307955] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13594.928966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13595.548901] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13600.230695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13600.236446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13600.262230] Lustre: Skipped 2 previous similar messages [13600.277030] Lustre: Skipped 2 previous similar messages [13602.139468] Lustre: server umount lustre-MDT0000 complete [13602.391797] LustreError: 424424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13604.780894] LustreError: 422970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.57@tcp arrived at 1789064133 with bad export cookie 17027826378171092633 [13607.431355] LustreError: 422970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064136 with bad export cookie 17027826378171091737 [13607.452441] LustreError: 422970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13607.463205] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13610.465608] LustreError: 422989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13610.467670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13610.492204] LustreError: 422989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13610.528266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13612.515168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13613.712317] Lustre: server umount lustre-OST0000 complete [13623.040131] Lustre: server umount lustre-MDT0001 complete [13640.194546] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [13644.205153] Key type lgssc unregistered [13644.608944] LNet: 426673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13644.617424] LNetError: 426673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13644.638838] LNet: Removed LNI 192.168.201.157@tcp [13645.775735] Key type .llcrypt unregistered [13645.778887] Key type ._llcrypt unregistered [13677.771979] Key type ._llcrypt registered [13677.774196] Key type .llcrypt registered [13677.914724] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [13679.069986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13679.113734] alg: No test for adler32 (adler32-zlib) [13680.326472] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [13680.662383] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [13682.368740] Key type lgssc registered [13683.539283] Lustre: Echo OBD driver; http://www.lustre.org/ [13693.357323] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13693.398284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13695.220641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13701.037907] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13705.700346] LustreError: 428086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13705.717392] LustreError: 428086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13710.354518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13710.710427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13716.364910] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13725.992503] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13734.702122] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13743.736369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13744.205280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13751.917461] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13761.783757] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13765.694423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13770.742698] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13770.780483] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13772.903253] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.57@tcp (stopping) [13772.916840] Lustre: Skipped 1 previous similar message [13775.852368] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13775.862731] Lustre: Skipped 1 previous similar message [13776.643195] Lustre: server umount lustre-MDT0001 complete [13777.968377] LustreError: 430400:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.57@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13780.962223] LustreError: 428086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13780.992517] LustreError: 428086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13786.080456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13786.084822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13786.086061] LustreError: 429031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13786.126425] Lustre: Skipped 1 previous similar message [13786.143268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13790.408326] Lustre: server umount lustre-OST0000 complete [13791.211916] LustreError: 428086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13796.230614] Lustre: server umount lustre-MDT0000 complete [13813.765683] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [13818.046795] Key type lgssc unregistered [13818.432869] LNet: 431776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13818.441204] LNetError: 431776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13818.456721] LNet: Removed LNI 192.168.201.157@tcp [13819.393782] Key type .llcrypt unregistered [13819.396377] Key type ._llcrypt unregistered [13849.925427] Key type ._llcrypt registered [13849.927500] Key type .llcrypt registered [13850.055833] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [13868.965456] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [13870.366568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13870.497772] alg: No test for adler32 (adler32-zlib) [13871.727800] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [13872.035586] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [13873.799307] Key type lgssc registered [13875.319147] Lustre: Echo OBD driver; http://www.lustre.org/ [13919.508027] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [13940.260409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13940.299162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13941.729633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13941.786474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13941.948233] Lustre: lustre-MDT0000: new disk, initializing [13942.103985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13942.138889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13947.490173] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13965.288681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13965.603553] Lustre: lustre-OST0000: new disk, initializing [13965.610321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13965.618768] Lustre: Skipped 1 previous similar message [13965.628847] Lustre: 436809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13965.702441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13972.510189] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [13974.111945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13974.122124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13974.147162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13982.759555] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13999.129178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13999.286341] Lustre: 435793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13999.441741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13999.549980] Lustre: lustre-MDT0001: new disk, initializing [13999.743751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13999.808489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13999.832991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14006.526389] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14018.555555] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14018.934742] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14025.407105] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14025.706689] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14031.845274] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14031.849140] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14031.849333] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14031.867937] Lustre: Skipped 1 previous similar message [14035.018790] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.57@tcp (stopping) [14035.025955] Lustre: Skipped 1 previous similar message [14036.966193] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14036.973220] Lustre: Skipped 1 previous similar message [14037.500349] Lustre: server umount lustre-MDT0001 complete [14047.205525] LustreError: 438944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14047.227025] LustreError: 438944:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [14052.334770] LustreError: 435799:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14052.356732] LustreError: 435799:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14057.454360] LustreError: 435803:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14057.494721] LustreError: 435803:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14061.026500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14061.045187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14061.064766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14062.564626] LustreError: 435802:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14064.840355] Lustre: server umount lustre-OST0000 complete [14067.695711] LustreError: 435803:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14076.972268] Lustre: server umount lustre-MDT0000 complete [14094.610623] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [14098.217941] Key type lgssc unregistered [14098.713970] LNet: 440559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14098.727075] LNetError: 440559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14098.753712] LNet: Removed LNI 192.168.201.157@tcp [14100.396411] Key type .llcrypt unregistered [14100.398231] Key type ._llcrypt unregistered [14154.605843] Key type ._llcrypt registered [14154.608883] Key type .llcrypt registered [14154.839879] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [14155.491337] Key type .llcrypt unregistered [14155.493674] Key type ._llcrypt unregistered [14176.278836] Key type ._llcrypt registered [14176.280386] Key type .llcrypt registered [14176.408235] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [14178.116127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14178.275840] alg: No test for adler32 (adler32-zlib) [14179.543106] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [14179.900433] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [14181.583245] Key type lgssc registered [14183.016831] Lustre: Echo OBD driver; http://www.lustre.org/ [14216.418710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14216.449442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14217.759613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14217.789960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14217.922128] Lustre: lustre-MDT0000: new disk, initializing [14218.010329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14218.025298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14223.617481] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14237.044108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14237.218978] Lustre: 443933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14237.375239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14237.381077] Lustre: Skipped 1 previous similar message [14237.575973] Lustre: lustre-MDT0001: new disk, initializing [14237.780633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14237.831636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14237.860372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14243.761692] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14255.584624] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14262.038792] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14270.787325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14271.309673] Lustre: lustre-OST0000: new disk, initializing [14271.312332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14271.317488] Lustre: 445902:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14271.397957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14277.807098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14277.820251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14277.989476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14279.693345] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14291.491892] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14303.716796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14303.723948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14303.743050] Lustre: Skipped 1 previous similar message [14303.771540] Lustre: Skipped 2 previous similar messages [14306.645565] Lustre: server umount lustre-MDT0000 complete [14311.496980] LustreError: 444737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064840 with bad export cookie 16847989145562978274 [14311.510057] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14311.517684] LustreError: 444737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14312.195899] Lustre: server umount lustre-MDT0001 complete [14327.643979] Lustre: server umount lustre-OST0000 complete [14340.798941] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:27:49 (1789064869) [14350.330513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14350.870806] LustreError: 447933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14350.891180] LustreError: 447933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14350.984345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14356.450055] LustreError: 447934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14356.735892] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14361.580477] LustreError: 447933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14366.705805] LustreError: 447934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14367.333448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14367.923896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14374.102751] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14383.470683] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14391.159984] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14412.268686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14412.560543] Lustre: Found index 0 for lustre-OST0000, updating log [14412.861146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14420.618432] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [14422.084949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14429.332736] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14437.357188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14437.367903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14437.390892] Lustre: Skipped 1 previous similar message [14437.405898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14438.442067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14441.799150] Lustre: server umount lustre-OST0000 complete [14459.036363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14459.073746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14459.089303] Lustre: Skipped 2 previous similar messages [14464.229508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14464.238375] Lustre: Skipped 1 previous similar message [14465.560594] Lustre: server umount lustre-MDT0000 complete [14470.346684] LustreError: 447913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789064999 with bad export cookie 16847989145562979667 [14470.353467] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14470.367856] LustreError: 447913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [14471.027100] Lustre: server umount lustre-MDT0001 complete [14480.995967] LNet: 1 local NIs in recovery (showing 1): 192.168.201.157@tcp [14481.489642] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:30:10 (1789065010) [15111.495068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15111.736980] LustreError: 454344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15111.785438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15117.282022] LustreError: 454345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15117.483727] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15121.377878] LustreError: 454344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15126.736217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15127.252933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15133.082609] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15141.902408] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15148.341220] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15169.071784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15169.454408] Lustre: Found index 0 for lustre-OST0000, updating log [15169.821123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15171.170858] LNet: 1 local NIs in recovery (showing 1): 192.168.201.157@tcp [15177.102754] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15185.072974] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15188.135751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15188.156388] Lustre: Skipped 1 previous similar message [15188.160977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15188.170081] Lustre: Skipped 2 previous similar messages [15193.444818] Lustre: server umount lustre-OST0000 complete [15197.261251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15197.269095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15197.276657] Lustre: Skipped 1 previous similar message [15197.299373] Lustre: Skipped 1 previous similar message [15199.335040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15199.350419] Lustre: Skipped 2 previous similar messages [15203.426586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15203.432212] Lustre: Skipped 7 previous similar messages [15204.185607] Lustre: server umount lustre-MDT0000 complete [15204.457203] LustreError: 454344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15204.486803] LustreError: 454344:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15208.558967] LustreError: 454344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15208.584881] LustreError: 454344:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [15208.733628] LustreError: 454326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789065737 with bad export cookie 16847989145562980738 [15208.747304] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15208.754888] LustreError: 454326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15209.192365] Lustre: server umount lustre-MDT0001 complete [15231.585416] LNet: 1 local NIs in recovery (showing 1): 192.168.201.157@tcp [15241.801993] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [15253.081580] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [15298.999634] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:43:47 (1789065827) [15317.926913] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [15320.862113] Key type lgssc unregistered [15321.208671] LNet: 460957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15321.220421] LNetError: 460957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15430.831021] LNet: Removed LNI 192.168.201.157@tcp [15432.243279] Key type .llcrypt unregistered [15432.245521] Key type ._llcrypt unregistered [15451.969681] Key type ._llcrypt registered [15451.971101] Key type .llcrypt registered [15452.044047] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [15453.390920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15453.495641] alg: No test for adler32 (adler32-zlib) [15454.842717] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [15455.126042] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [15456.903225] Key type lgssc registered [15458.519557] Lustre: Echo OBD driver; http://www.lustre.org/ [15462.718830] LNet: Added LNI 192.168.201.157@tcp100 [8/256/0/180] [15463.518792] LNet: Added LNI 192.168.201.157@tcp101 [8/256/0/180] [15464.480894] LNet: Added LNI 192.168.201.157@tcp102 [8/256/0/180] [15465.418155] LNet: Added LNI 192.168.201.157@tcp103 [8/256/0/180] [15466.388986] LNet: Added LNI 192.168.201.157@tcp104 [8/256/0/180] [15467.201285] LNet: Added LNI 192.168.201.157@tcp105 [8/256/0/180] [15468.230895] LNet: Added LNI 192.168.201.157@tcp106 [8/256/0/180] [15469.343325] LNet: Added LNI 192.168.201.157@tcp107 [8/256/0/180] [15470.331596] LNet: Added LNI 192.168.201.157@tcp108 [8/256/0/180] [15471.480082] LNet: Added LNI 192.168.201.157@tcp109 [8/256/0/180] [15472.511150] LNet: Added LNI 192.168.201.157@tcp110 [8/256/0/180] [15473.300932] LNet: Added LNI 192.168.201.157@tcp111 [8/256/0/180] [15474.165870] LNet: Added LNI 192.168.201.157@tcp112 [8/256/0/180] [15474.971316] LNet: Added LNI 192.168.201.157@tcp113 [8/256/0/180] [15475.837945] LNet: Added LNI 192.168.201.157@tcp114 [8/256/0/180] [15476.598759] LNet: Added LNI 192.168.201.157@tcp115 [8/256/0/180] [15477.654860] LNet: Added LNI 192.168.201.157@tcp116 [8/256/0/180] [15478.591342] LNet: Added LNI 192.168.201.157@tcp117 [8/256/0/180] [15479.443302] LNet: Added LNI 192.168.201.157@tcp118 [8/256/0/180] [15480.286387] LNet: Added LNI 192.168.201.157@tcp119 [8/256/0/180] [15481.142504] LNet: Added LNI 192.168.201.157@tcp120 [8/256/0/180] [15481.930331] LNet: Added LNI 192.168.201.157@tcp121 [8/256/0/180] [15482.702449] LNet: Added LNI 192.168.201.157@tcp122 [8/256/0/180] [15483.586102] LNet: Added LNI 192.168.201.157@tcp123 [8/256/0/180] [15484.472934] LNet: Added LNI 192.168.201.157@tcp124 [8/256/0/180] [15485.520710] LNet: Added LNI 192.168.201.157@tcp125 [8/256/0/180] [15486.335121] LNet: Added LNI 192.168.201.157@tcp126 [8/256/0/180] [15487.406647] LNet: Added LNI 192.168.201.157@tcp127 [8/256/0/180] [15488.416860] LNet: Added LNI 192.168.201.157@tcp128 [8/256/0/180] [15489.403698] LNet: Added LNI 192.168.201.157@tcp129 [8/256/0/180] [15490.532752] LNet: Added LNI 192.168.201.157@tcp130 [8/256/0/180] [15491.469849] LNet: Added LNI 192.168.201.157@tcp131 [8/256/0/180] [15492.346704] LNet: Added LNI 192.168.201.157@tcp132 [8/256/0/180] [15493.354742] LNet: Added LNI 192.168.201.157@tcp133 [8/256/0/180] [15494.377906] LNet: Added LNI 192.168.201.157@tcp134 [8/256/0/180] [15494.966285] LNet: Added LNI 192.168.201.157@tcp135 [8/256/0/180] [15503.074573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15503.103366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15504.331952] Lustre: 463985:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.157@tcp: can't add NID 192.168.201.157@tcp130: rc = -75 [15504.469584] LNetError: 461442:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [15504.487481] LNetError: Protocol error connecting to 192.168.201.157@tcp118 at host 192.168.201.157:988: is it running a compatible version of Lustre? [15504.517931] LNetError: Skipped 4 previous similar messages [15504.579069] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15504.598960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15504.666563] Lustre: lustre-MDT0000: new disk, initializing [15504.733947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15504.770313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15505.312072] LNet: 8 local NIs in recovery (showing 8): 192.168.201.157@tcp121, 192.168.201.157@tcp123, 192.168.201.157@tcp122, 192.168.201.157@tcp125, 192.168.201.157@tcp126, 192.168.201.157@tcp127, 192.168.201.157@tcp118, 192.168.201.157@tcp129 [15505.313665] LNetError: 461445:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [15505.363483] LNetError: 461445:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [15505.431625] LNetError: Protocol error connecting to 192.168.201.157@tcp116 at host 192.168.201.157:988: is it running a compatible version of Lustre? [15505.458078] LNetError: Skipped 6 previous similar messages [15506.469277] LNetError: 461444:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [15506.490714] LNetError: 461444:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15506.521685] LNetError: Protocol error connecting to 192.168.201.157@tcp129 at host 192.168.201.157:988: is it running a compatible version of Lustre? [15511.864511] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15525.084138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15525.172919] Lustre: 464001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15525.227345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15525.228812] Lustre: Skipped 1 previous similar message [15525.306217] Lustre: lustre-MDT0001: new disk, initializing [15525.329443] Lustre: 464804:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.157@tcp: can't add NID 192.168.201.157@tcp130: rc = -75 [15525.333986] Lustre: 464804:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15525.474368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15525.518350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15525.533194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15530.078479] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15530.528697] Lustre: 465086:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.157@tcp: can't add NID 192.168.201.157@tcp130: rc = -75 [15530.540967] Lustre: 465086:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15534.179800] LNetError: 464015:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [15534.186212] LNetError: Protocol error connecting to 192.168.201.157@tcp119 at host 192.168.201.157:988: is it running a compatible version of Lustre? [15538.284480] LNetError: 464017:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [15538.305544] LNetError: Protocol error connecting to 192.168.201.157@tcp125 at host 192.168.201.157:988: is it running a compatible version of Lustre? [15541.628739] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15546.468442] LNetError: 461443:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [15546.517973] LNetError: 461443:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15546.547317] LNetError: Protocol error connecting to 192.168.201.157@tcp124 at host 192.168.201.157:988: is it running a compatible version of Lustre? [15546.572150] LNetError: Skipped 3 previous similar messages [15547.290628] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15555.504943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15556.012700] Lustre: lustre-OST0000: new disk, initializing [15556.016465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15556.022533] Lustre: 465979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15556.136510] Lustre: 465984:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.157@tcp: can't add NID 192.168.201.157@tcp130: rc = -75 [15556.153902] Lustre: 465984:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15556.243498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15562.847701] Lustre: 466308:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.157@tcp: can't add NID 192.168.201.157@tcp130: rc = -75 [15562.861789] Lustre: 466308:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15562.933785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15562.952655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15563.127240] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15563.157419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15563.879344] LNetError: 461442:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [15563.892925] LNetError: 461442:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15563.905348] LNetError: Protocol error connecting to 192.168.201.157@tcp125 at host 192.168.201.157:988: is it running a compatible version of Lustre? [15563.919724] LNetError: Skipped 7 previous similar messages [15573.800390] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15588.833683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15588.838417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15588.842245] Lustre: Skipped 1 previous similar message [15588.844296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15588.866286] Lustre: Skipped 1 previous similar message [15590.510722] Lustre: server umount lustre-OST0000 complete [15596.640313] LNetError: 461443:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.157 [15596.647655] LNetError: 461443:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15596.651700] LNetError: Protocol error connecting to 192.168.201.157@tcp116 at host 192.168.201.157:988: is it running a compatible version of Lustre? [15596.657790] LNetError: Skipped 9 previous similar messages [15597.030054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15597.034713] LustreError: Skipped 1 previous similar message [15597.037204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15597.045424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15599.077175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15599.080827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15603.121890] Lustre: server umount lustre-MDT0000 complete [15607.145258] LustreError: 463993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066136 with bad export cookie 9262818084777356414 [15607.146755] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15607.153099] LustreError: 463993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15607.525429] Lustre: server umount lustre-MDT0001 complete [15621.214185] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [15624.276874] Key type lgssc unregistered [15624.732712] LNet: 467895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15624.739608] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15624.747645] LNet: Removed LNI 192.168.201.157@tcp [15625.759394] LNet: Removed LNI 192.168.201.157@tcp100 [15625.775792] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15625.788198] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15625.801658] LNet: Removed LNI 192.168.201.157@tcp101 [15626.847599] LNet: Removed LNI 192.168.201.157@tcp102 [15626.854208] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15626.865151] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15626.874738] LNet: Removed LNI 192.168.201.157@tcp103 [15627.938124] LNet: Removed LNI 192.168.201.157@tcp104 [15628.959183] LNet: Removed LNI 192.168.201.157@tcp105 [15628.969551] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15628.977196] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15628.995153] LNet: Removed LNI 192.168.201.157@tcp106 [15630.047350] LNet: Removed LNI 192.168.201.157@tcp107 [15631.071428] LNet: Removed LNI 192.168.201.157@tcp108 [15632.096749] LNet: Removed LNI 192.168.201.157@tcp109 [15633.119282] LNet: Removed LNI 192.168.201.157@tcp110 [15633.145108] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15633.172159] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15633.183359] LNet: Removed LNI 192.168.201.157@tcp111 [15634.208361] LNet: Removed LNI 192.168.201.157@tcp112 [15635.231211] LNet: Removed LNI 192.168.201.157@tcp113 [15636.255167] LNet: Removed LNI 192.168.201.157@tcp114 [15637.279168] LNet: Removed LNI 192.168.201.157@tcp115 [15638.303202] LNet: Removed LNI 192.168.201.157@tcp116 [15639.327406] LNet: Removed LNI 192.168.201.157@tcp117 [15640.351211] LNet: Removed LNI 192.168.201.157@tcp118 [15641.375263] LNet: Removed LNI 192.168.201.157@tcp119 [15641.383332] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15641.400909] LNetError: 467895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15641.414905] LNet: Removed LNI 192.168.201.157@tcp120 [15642.463256] LNet: Removed LNI 192.168.201.157@tcp121 [15643.487226] LNet: Removed LNI 192.168.201.157@tcp122 [15644.511166] LNet: Removed LNI 192.168.201.157@tcp123 [15645.535205] LNet: Removed LNI 192.168.201.157@tcp124 [15645.551938] LNet: Removed LNI 192.168.201.157@tcp125 [15646.623462] LNet: Removed LNI 192.168.201.157@tcp126 [15647.711498] LNet: Removed LNI 192.168.201.157@tcp127 [15648.735308] LNet: Removed LNI 192.168.201.157@tcp128 [15649.759284] LNet: Removed LNI 192.168.201.157@tcp129 [15649.765512] LNet: Removed LNI 192.168.201.157@tcp130 [15649.768361] LNet: Removed LNI 192.168.201.157@tcp131 [15649.777239] LNet: Removed LNI 192.168.201.157@tcp132 [15649.790902] LNet: Removed LNI 192.168.201.157@tcp133 [15649.796894] LNet: Removed LNI 192.168.201.157@tcp134 [15649.823811] LNet: Removed LNI 192.168.201.157@tcp135 [15651.039280] Key type .llcrypt unregistered [15651.043468] Key type ._llcrypt unregistered [15682.993851] Key type ._llcrypt registered [15683.001909] Key type .llcrypt registered [15683.146391] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [15701.962801] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [15704.057896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15704.114598] alg: No test for adler32 (adler32-zlib) [15705.358802] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [15705.710262] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [15707.480153] Key type lgssc registered [15708.888879] Lustre: Echo OBD driver; http://www.lustre.org/ [15748.073422] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:51:16 (1789066276) [15768.278759] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [15772.463487] Key type lgssc unregistered [15772.901579] LNet: 471981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15772.907136] LNetError: 471981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15772.924636] LNet: Removed LNI 192.168.201.157@tcp [15773.837443] Key type .llcrypt unregistered [15773.839270] Key type ._llcrypt unregistered [15798.179781] Key type ._llcrypt registered [15798.182440] Key type .llcrypt registered [15798.373368] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [15800.069662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15800.135026] alg: No test for adler32 (adler32-zlib) [15801.393681] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [15801.664610] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [15803.327276] Key type lgssc registered [15804.326331] Lustre: Echo OBD driver; http://www.lustre.org/ [15815.939948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15816.050890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15817.692847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15817.773461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15817.956891] Lustre: lustre-MDT0000: new disk, initializing [15818.082877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15818.145879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15825.017979] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15839.446642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15839.553621] Lustre: 473193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15839.609905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15839.614789] Lustre: Skipped 1 previous similar message [15839.666937] Lustre: lustre-MDT0001: new disk, initializing [15839.755744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15839.826013] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15839.844646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15845.199244] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15856.735335] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15862.048928] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15868.262840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15868.437665] Lustre: lustre-OST0000: new disk, initializing [15868.439755] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15868.443926] Lustre: 475162:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15868.506107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15874.693638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15874.718583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15874.950196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15877.649623] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [15889.082900] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15894.150399] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing lnet_if_list [15900.721828] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15900.779146] LNet: Added LNI 192.168.201.157@tcp10 [8/256/0/180] [15906.327678] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15906.340813] LNet: Added LNI 192.168.201.157@tcp11 [8/256/0/180] [15909.543144] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15909.557900] LNet: Added LNI 192.168.201.157@tcp12 [8/256/0/180] [15915.191839] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15915.221315] LNet: Added LNI 192.168.201.157@tcp13 [8/256/0/180] [15920.579925] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15920.610677] LNet: Added LNI 192.168.201.157@tcp14 [8/256/0/180] [15926.695908] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15926.722063] LNet: Added LNI 192.168.201.157@tcp15 [8/256/0/180] [15932.367687] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15932.391812] LNet: Added LNI 192.168.201.157@tcp16 [8/256/0/180] [15937.666552] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15937.690809] LNet: Added LNI 192.168.201.157@tcp17 [8/256/0/180] [15943.909711] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15943.976928] LNet: Added LNI 192.168.201.157@tcp18 [8/256/0/180] [15951.721409] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15951.772960] LNet: Added LNI 192.168.201.157@tcp19 [8/256/0/180] [15960.162486] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15960.247389] LNet: Added LNI 192.168.201.157@tcp20 [8/256/0/180] [15966.753939] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15966.830728] LNet: Added LNI 192.168.201.157@tcp21 [8/256/0/180] [15975.012969] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15975.041469] LNet: Added LNI 192.168.201.157@tcp22 [8/256/0/180] [15982.526630] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15982.599734] LNet: Added LNI 192.168.201.157@tcp23 [8/256/0/180] [15989.741849] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15989.772929] LNet: Added LNI 192.168.201.157@tcp24 [8/256/0/180] [15995.805922] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15995.831278] LNet: Added LNI 192.168.201.157@tcp25 [8/256/0/180] [16002.985631] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16003.012329] LNet: Added LNI 192.168.201.157@tcp26 [8/256/0/180] [16010.243600] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16010.295233] LNet: Added LNI 192.168.201.157@tcp27 [8/256/0/180] [16017.584698] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16017.634539] LNet: Added LNI 192.168.201.157@tcp28 [8/256/0/180] [16024.782535] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16024.827767] LNet: Added LNI 192.168.201.157@tcp29 [8/256/0/180] [16032.700234] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16032.751165] LNet: Added LNI 192.168.201.157@tcp30 [8/256/0/180] [16041.163990] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16041.217782] LNet: Added LNI 192.168.201.157@tcp31 [8/256/0/180] [16049.831927] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16049.892320] LNet: Added LNI 192.168.201.157@tcp32 [8/256/0/180] [16058.442684] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16058.464047] LNet: Added LNI 192.168.201.157@tcp33 [8/256/0/180] [16066.892463] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16066.936509] LNet: Added LNI 192.168.201.157@tcp34 [8/256/0/180] [16074.326776] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16074.358964] LNet: Added LNI 192.168.201.157@tcp35 [8/256/0/180] [16081.946488] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16081.989899] LNet: Added LNI 192.168.201.157@tcp36 [8/256/0/180] [16091.179445] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16091.218565] LNet: Added LNI 192.168.201.157@tcp37 [8/256/0/180] [16099.685542] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16099.729258] LNet: Added LNI 192.168.201.157@tcp38 [8/256/0/180] [16109.403970] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16109.439686] LNet: Added LNI 192.168.201.157@tcp39 [8/256/0/180] [16119.305119] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16119.359257] LNet: Added LNI 192.168.201.157@tcp40 [8/256/0/180] [16130.181196] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16130.224062] LNet: Added LNI 192.168.201.157@tcp41 [8/256/0/180] [16140.064963] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16140.091877] LNet: Added LNI 192.168.201.157@tcp42 [8/256/0/180] [16148.407739] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16148.439720] LNet: Added LNI 192.168.201.157@tcp43 [8/256/0/180] [16156.470939] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16156.503064] LNet: Added LNI 192.168.201.157@tcp44 [8/256/0/180] [16166.762284] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16166.832244] LNet: Added LNI 192.168.201.157@tcp45 [8/256/0/180] [16178.147749] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16178.191084] LNet: Added LNI 192.168.201.157@tcp46 [8/256/0/180] [16189.325450] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16189.374987] LNet: Added LNI 192.168.201.157@tcp47 [8/256/0/180] [16199.962854] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16199.988432] LNet: Added LNI 192.168.201.157@tcp48 [8/256/0/180] [16209.581148] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16209.632765] LNet: Added LNI 192.168.201.157@tcp49 [8/256/0/180] [16219.823299] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16219.875844] LNet: Added LNI 192.168.201.157@tcp50 [8/256/0/180] [16243.700887] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16243.703031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16243.729680] Lustre: Skipped 2 previous similar messages [16243.739949] Lustre: Skipped 1 previous similar message [16248.804364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16248.815094] Lustre: Skipped 1 previous similar message [16249.121606] Lustre: server umount lustre-MDT0000 complete [16253.005286] LustreError: 473184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066782 with bad export cookie 8008280830763202875 [16253.009521] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16253.031737] LustreError: 473184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16253.572326] Lustre: server umount lustre-MDT0001 complete [16270.303164] Lustre: 472462:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789066783/real 1789066783] req@00000000737cc387 x1875972016240256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789066799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16270.335442] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16275.424339] Lustre: 472463:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789066788/real 1789066788] req@00000000e33241e9 x1875972016240384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789066804 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16276.498815] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [16279.527845] Lustre: 472465:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789066792/real 1789066792] req@0000000023d76c2c x1875972016240640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789066808 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16284.707816] Lustre: 472465:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789066797/real 1789066797] req@000000005e93d898 x1875972016240768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789066813 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16287.724590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16298.608917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16299.620473] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16303.985104] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16306.147778] LustreError: 475155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16306.163964] LustreError: 475155:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16311.276825] LustreError: 475155:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16314.714993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16315.337267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16316.323865] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16320.530738] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16329.121825] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16336.459446] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16359.907594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16359.911887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16359.943113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16361.453203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16362.465984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16362.475345] Lustre: Skipped 2 previous similar messages [16365.799182] Lustre: server umount lustre-OST0000 complete [16371.684103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16371.718229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16371.762183] Lustre: Skipped 1 previous similar message [16371.784584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16376.906301] Lustre: server umount lustre-MDT0000 complete [16377.830503] LustreError: 499433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16377.848850] LustreError: 499433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16380.898955] LustreError: 473184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789066910 with bad export cookie 8008280830763205549 [16380.905668] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16380.919610] LustreError: 473184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16381.769990] Lustre: server umount lustre-MDT0001 complete [16395.215743] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [16398.608798] Key type lgssc unregistered [16399.009922] LNet: 502242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16399.035177] LNetError: 502242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16400.095182] LNet: Removed LNI 192.168.201.157@tcp [16400.096926] LNetError: 502242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16400.103990] LNet: Removed LNI 192.168.201.157@tcp10 [16400.111315] LNet: Removed LNI 192.168.201.157@tcp11 [16400.116549] LNet: Removed LNI 192.168.201.157@tcp12 [16400.124763] LNet: Removed LNI 192.168.201.157@tcp13 [16400.127668] LNet: Removed LNI 192.168.201.157@tcp14 [16400.137701] LNet: Removed LNI 192.168.201.157@tcp15 [16400.142443] LNet: Removed LNI 192.168.201.157@tcp16 [16400.147929] LNet: Removed LNI 192.168.201.157@tcp17 [16400.154105] LNet: Removed LNI 192.168.201.157@tcp18 [16400.158703] LNet: Removed LNI 192.168.201.157@tcp19 [16400.165507] LNet: Removed LNI 192.168.201.157@tcp20 [16400.171301] LNet: Removed LNI 192.168.201.157@tcp21 [16400.174674] LNet: Removed LNI 192.168.201.157@tcp22 [16400.179320] LNet: Removed LNI 192.168.201.157@tcp23 [16400.182497] LNet: Removed LNI 192.168.201.157@tcp24 [16400.185273] LNet: Removed LNI 192.168.201.157@tcp25 [16400.190905] LNet: Removed LNI 192.168.201.157@tcp26 [16400.194673] LNet: Removed LNI 192.168.201.157@tcp27 [16400.197929] LNet: Removed LNI 192.168.201.157@tcp28 [16400.200727] LNet: Removed LNI 192.168.201.157@tcp29 [16400.203911] LNet: Removed LNI 192.168.201.157@tcp30 [16400.207428] LNet: Removed LNI 192.168.201.157@tcp31 [16400.209682] LNet: Removed LNI 192.168.201.157@tcp32 [16400.214047] LNet: Removed LNI 192.168.201.157@tcp33 [16400.217323] LNet: Removed LNI 192.168.201.157@tcp34 [16400.222960] LNet: Removed LNI 192.168.201.157@tcp35 [16400.225484] LNet: Removed LNI 192.168.201.157@tcp36 [16400.227793] LNet: Removed LNI 192.168.201.157@tcp37 [16400.230523] LNet: Removed LNI 192.168.201.157@tcp38 [16400.233505] LNet: Removed LNI 192.168.201.157@tcp39 [16400.236050] LNet: Removed LNI 192.168.201.157@tcp40 [16400.238323] LNet: Removed LNI 192.168.201.157@tcp41 [16400.240474] LNet: Removed LNI 192.168.201.157@tcp42 [16400.243147] LNet: Removed LNI 192.168.201.157@tcp43 [16400.245917] LNet: Removed LNI 192.168.201.157@tcp44 [16400.251878] LNet: Removed LNI 192.168.201.157@tcp45 [16400.256134] LNet: Removed LNI 192.168.201.157@tcp46 [16400.257345] LNet: Removed LNI 192.168.201.157@tcp47 [16400.262260] LNet: Removed LNI 192.168.201.157@tcp48 [16400.264977] LNet: Removed LNI 192.168.201.157@tcp49 [16400.273850] LNet: Removed LNI 192.168.201.157@tcp50 [16401.052189] Key type .llcrypt unregistered [16401.053391] Key type ._llcrypt unregistered [16424.207111] Key type ._llcrypt registered [16424.208368] Key type .llcrypt registered [16424.318814] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [16441.318362] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [16443.114585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16443.159573] alg: No test for adler32 (adler32-zlib) [16444.285717] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [16444.619628] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [16446.368750] Key type lgssc registered [16447.851763] Lustre: Echo OBD driver; http://www.lustre.org/ [16489.353552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16489.393864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16490.836105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16490.896414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16491.024334] Lustre: lustre-MDT0000: new disk, initializing [16491.153080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16491.193067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16496.939475] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16509.138367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16509.352867] Lustre: 505451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16509.454335] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16509.463896] Lustre: Skipped 1 previous similar message [16509.590766] Lustre: lustre-MDT0001: new disk, initializing [16509.715109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16509.758539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16509.783780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16515.338354] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16525.567596] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16531.723550] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16540.620353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16540.876258] Lustre: *** cfs_fail_loc=90f, val=0*** [16541.005484] Lustre: lustre-OST0000: new disk, initializing [16541.021872] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16541.031471] Lustre: 507471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16541.158505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16548.680794] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16549.480724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16549.515779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16549.686469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16560.087747] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16570.342461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16570.344826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16570.346355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16570.388033] LustreError: Skipped 1 previous similar message [16573.731397] Lustre: server umount lustre-OST0000 complete [16580.585159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16580.590666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16580.605106] Lustre: Skipped 2 previous similar messages [16580.615369] Lustre: Skipped 1 previous similar message [16583.740000] Lustre: server umount lustre-MDT0000 complete [16587.711250] LustreError: 505443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067116 with bad export cookie 17754333310499053767 [16587.721169] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16587.733586] LustreError: 505443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16588.085774] Lustre: server umount lustre-MDT0001 complete [16600.895694] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [16604.416859] Key type lgssc unregistered [16604.734660] LNet: 509376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16604.751136] LNetError: 509376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16604.780055] LNet: Removed LNI 192.168.201.157@tcp [16605.860660] Key type .llcrypt unregistered [16605.864071] Key type ._llcrypt unregistered [16627.804540] Key type ._llcrypt registered [16627.807831] Key type .llcrypt registered [16627.931629] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [16629.465804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16629.546163] alg: No test for adler32 (adler32-zlib) [16630.979975] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [16631.347442] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [16633.159267] Key type lgssc registered [16634.580977] Lustre: Echo OBD driver; http://www.lustre.org/ [16643.236961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16643.251061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16644.816721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16650.119987] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16654.307341] LustreError: 510642:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16654.334041] LustreError: 510642:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16659.425979] LustreError: 510643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16659.487359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16659.791938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16664.755406] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16675.605176] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16683.539708] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16690.635579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16690.985130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16699.785813] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16708.226878] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16725.670222] loop0: detected capacity change from 0 to 204800000 [16725.958934] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16725.992462] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16726.032852] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16732.919764] loop0: detected capacity change from 0 to 204800000 [16733.406333] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16733.427775] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16733.455511] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16739.197789] loop0: detected capacity change from 0 to 204800000 [16744.334039] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16744.603373] Lustre: fs15246-OST0000: new disk, initializing [16744.608887] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16744.621317] Lustre: 514260:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16744.699838] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16751.673567] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16763.568948] loop1: detected capacity change from 0 to 204800000 [16770.461227] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16770.624962] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16770.734419] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16770.827930] Lustre: fs15246-MDT0000: new disk, initializing [16771.042770] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16771.068607] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16771.231763] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16772.255984] Lustre: 515382:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d909593c x1875972886415872/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789067317 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16776.556454] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [16781.353087] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16781.369408] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16781.402710] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16800.738658] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16800.753180] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16805.859579] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16806.172611] Lustre: server umount fs15246-MDT0000 complete [16810.989462] LustreError: 513134:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16816.111992] LustreError: 513134:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16816.493531] Lustre: server umount fs15246-OST0000 complete [16831.465066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16831.469075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16831.480693] Lustre: Skipped 1 previous similar message [16831.490990] Lustre: Skipped 1 previous similar message [16835.557862] Lustre: server umount lustre-OST0000 complete [16841.698795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16841.710592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16841.715194] Lustre: Skipped 1 previous similar message [16841.743236] Lustre: Skipped 1 previous similar message [16846.736148] Lustre: server umount lustre-MDT0000 complete [16846.819827] LustreError: 510639:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16846.848491] LustreError: 510639:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16852.463790] LustreError: 510624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067381 with bad export cookie 10817234260323414663 [16852.469538] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16852.472049] LustreError: 510624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16853.212694] Lustre: server umount lustre-MDT0001 complete [16868.720556] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [16872.770519] Key type lgssc unregistered [16873.182534] LNet: 518477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16873.186313] LNetError: 518477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16873.199991] LNet: Removed LNI 192.168.201.157@tcp [16874.343047] Key type .llcrypt unregistered [16874.344651] Key type ._llcrypt unregistered [16894.415547] Key type ._llcrypt registered [16894.418287] Key type .llcrypt registered [16894.537242] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [16895.778933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16895.807143] alg: No test for adler32 (adler32-zlib) [16896.885895] Lustre: Lustre: Build Version: 2.17.58_39_g3ed9deb [16897.113917] LNet: Added LNI 192.168.201.157@tcp [8/256/0/180] [16898.831590] Key type lgssc registered [16900.255582] Lustre: Echo OBD driver; http://www.lustre.org/ [16939.929372] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [16950.571974] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [16990.585468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16990.606650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16991.817917] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16991.848494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16991.932774] Lustre: lustre-MDT0000: new disk, initializing [16992.051856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16992.081768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16996.771919] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [17007.869110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17007.996686] Lustre: 522941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17008.088955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17008.094880] Lustre: Skipped 1 previous similar message [17008.227396] Lustre: lustre-MDT0001: new disk, initializing [17008.337898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17008.399903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17008.415637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17013.973178] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [17024.963726] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17031.125671] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17038.270136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17038.502173] Lustre: lustre-OST0000: new disk, initializing [17038.506336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17038.512528] Lustre: 524908:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17038.624904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17045.177925] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_default_debug -1 all [17048.195354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17048.201490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17048.320815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17056.452209] Lustre: DEBUG MARKER: oleg157-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17062.393679] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17062.622511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17067.746481] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17068.032795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17074.159967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17074.166826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17074.167671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17074.181198] Lustre: Skipped 1 previous similar message [17074.216672] LustreError: Skipped 1 previous similar message [17079.279268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17079.288833] Lustre: Skipped 2 previous similar messages [17084.384224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17084.388721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17084.424583] Lustre: Skipped 1 previous similar message [17084.649895] Lustre: server umount lustre-OST0000 complete [17089.504604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17089.516065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17094.628136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17094.635761] Lustre: Skipped 1 previous similar message [17095.756736] Lustre: server umount lustre-MDT0000 complete [17099.819872] LustreError: 522932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789067628 with bad export cookie 11766601503897544817 [17099.825613] LustreError: MGC192.168.201.157@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17099.830965] LustreError: 522932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17100.215339] Lustre: server umount lustre-MDT0001 complete [17127.805385] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing set_hostid [17141.948916] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing load_modules_local [17177.277955] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16873 sec ========= 15:15:05 (1789067705) [17179.095403] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:15:07 (1789067707) === [17183.034408] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:15:11 (1789067711) === [17227.806972] Lustre: DEBUG MARKER: oleg157-server.virtnet: executing unload_modules_local [17231.448509] Key type lgssc unregistered [17231.926766] LNet: 531556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17231.935306] LNetError: 531556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17231.965777] LNet: Removed LNI 192.168.201.157@tcp [17233.327513] Key type .llcrypt unregistered [17233.332436] Key type ._llcrypt unregistered